You do not need a graphics card to run a useful language model. A modern laptop or desktop CPU can chat, summarize and write code with a well-chosen model, privately and at no running cost. It just rewards a bit of understanding. Here is what we have learned building Matricxon, our CPU-first inference engine.
The one number that matters: memory bandwidth
When a model writes a reply, every new token means reading (nearly) all of the model’s weights from RAM once. The CPU’s arithmetic is rarely the limit; the memory bus is. A useful rule of thumb:
Tokens per second ≈ memory bandwidth ÷ model size in memory
A typical dual-channel DDR4 laptop moves around 40 GB/s, so a 4 GB model tops out near 10 tokens per second, and a 2 GB model near 20. Faster RAM, more memory channels and smaller models all help. Extra cores help much less than you would expect.
Reading your prompt is different: it is compute-bound, so long prompts and big documents take time on a CPU even when replies stream quickly. Keep that in mind when you paste a whole file into a chat.
Pick the right model
- Size first. On a laptop with 8 to 16 GB of RAM, models of 1 to 4 billion parameters feel responsive. A desktop with 32 GB or more handles 7 to 14B well.
- Quantize. GGUF files store weights in compressed blocks. Q4_K_M is the sweet spot for most models: about a quarter of the original size with little quality loss. Small models (under 3B) deserve Q5 or Q6, since every bit counts more for them. Below Q3, quality drops quickly.
- Consider mixture-of-experts. A model such as a 30B with 3B active parameters reads only the experts it uses for each token, so it can run nearly as fast as a 3B model while answering like a much larger one. It still needs RAM for the whole file.
Tune the server
- Threads: physical cores, not all of them. Once memory bandwidth is saturated, more threads add heat, not speed. On laptops, leaving one or two cores free keeps the server responsive and avoids thermal throttling, which can cost more speed than it saves.
- Never swap. A model that spills into swap slows to a crawl. Leave a safe margin of free RAM, and close the browser tab with forty other tabs in it.
- Fill both memory channels. One stick of RAM halves your bandwidth. Two matched sticks are one of the cheapest speed-ups there is.
- Keep the context modest. The chat’s memory (the KV cache) grows with every token of context. A 4K to 8K window is plenty for most chats and keeps RAM free for the model.
What Matricxon does for you
Matricxon is built around these limits, so most of this happens without any setup:
- Packed weights. By default, weights stay compressed in the memory-mapped file and are multiplied directly, with no unpacked copy in RAM. Ministral 3B in Q4_K_M needs about 2.6 GB this way instead of about 7.5 GB. If you have RAM to spare, set
MATRICXON_ENABLE_QUANTIZED_NATIVE_COMPUTE=false: unpacked weights are about 2 to 4 times faster per step. - Native C kernels. All the quantized math runs in C with OpenMP threads, for 25 GGUF quantization types, including the very small IQ types.
- Fewer threads, cooler server.
MATRICXON_TORCH_THREADScaps the thread count, for laptops that run hot. - Prompt caching. Each model keeps several conversations’ caches, and a new message reuses the longest matching prefix, so only the new text is read. Since reading prompts is the slow part on a CPU, this is the biggest win of all.
- Caches that survive. Caches evicted from RAM, or still held when the server stops, are saved to disk with AES encryption. Coming back to a long chat tomorrow does not mean re-reading it.
- Context shift. When a chat outgrows its window, the oldest tokens are dropped from the cache instead of re-reading everything.
- Safe loading. A model that would not fit in free RAM is refused rather than pushed into swap, and idle models unload after a while.
- Several users at once. Replies on one model run side by side, with their steps batched together where the architecture allows.
Quick checklist
- Choose a model that fits in RAM with room to spare, at Q4_K_M (Q5 or Q6 if small).
- Try a mixture-of-experts model if you have the RAM but want speed.
- Use your physical cores minus one or two.
- Keep chats going instead of starting over, so the prompt cache does its work.
- Measure: tokens per second tells you at once whether a change helped.
CPU inference will not beat a big GPU, but for private, everyday use it is often fast enough, and it runs on hardware you already own. To try it, start with the Matricxon quick start, or run it with pAIring for a full chat workspace.