“You need a GPU for AI” is half true. A graphics card makes some parts of running a language model much faster, and others barely at all. Knowing which is which saves money, and tells you whether the computer you already have is enough.
Both are limited by memory, not maths
When a model writes a reply, every new token means reading nearly all of its weights once. So the speed of writing is set mostly by how fast the chip can read its memory:
| Hardware | Memory speed (roughly) |
|---|---|
| Laptop or desktop, two DDR4/DDR5 channels | 40 to 90 GB/s |
| Workstation or server, many channels | 200 to 500 GB/s |
| Apple Silicon (unified memory) | 100 to 800 GB/s |
| Consumer graphics card | 300 to 1,000+ GB/s |
That is why a GPU often writes 5 to 15 times faster than a desktop CPU with the same model. It is not that it thinks harder; it reads faster.
The catch: the model has to fit
A GPU is only fast while the whole model, plus the chat’s memory (the KV cache), fits in its own memory (VRAM). Consumer cards have 8 to 24 GB. A 4-bit 8B model (about 5 GB) fits easily. A 4-bit 70B model (about 40 GB) does not fit on any consumer card, but runs on a computer with 64 GB of ordinary RAM.
Splitting a model between the GPU and the CPU works, but helps less than you might expect: every token waits for the slowest part, which is the half that sits in system memory.
Where the GPU really shines
- Reading the prompt. Before the first word of a reply, the model reads your whole message, and with a knowledge base, the documents found for it. This part is pure arithmetic, and a GPU can be 10 to 100 times faster. Long prompts are where a CPU feels slow.
- Many users at once. A GPU can handle several conversations in the same pass over the weights. A CPU mostly takes turns.
- Large models at speed. If you need a 30B to 70B model to answer quickly, you need serious GPU memory, or several cards.
Where the CPU holds its own
- Capacity. 64 GB of RAM costs a fraction of 64 GB of VRAM. Large models fit; they are just slower.
- Cost and what you already own. Every computer has a CPU. For one person, or a small team asking short questions, a well-chosen 3B to 8B model on a modern CPU is often fast enough.
- Mixture-of-experts models. They read only a small part of their weights for each token, so they run far faster on a CPU than their size suggests.
- Simplicity, power and noise. No drivers, no fans at full speed, lower power use, and it runs on any physical or virtual server.
Privacy is the same on both: a model on your own hardware keeps your data on your own hardware, whatever chip runs it.
How to choose
| Your situation | A good fit |
|---|---|
| One user, short questions, a recent laptop | CPU, a 3B to 8B model at 4-bit |
| A small team, long documents or a knowledge base | A GPU with 12 to 24 GB, or a fast multi-channel CPU |
| Large models, but speed is not critical | CPU with plenty of RAM, or a mixture-of-experts model |
| Large models, fast, many users | One or more GPUs with enough memory for the whole model |
| Occasional need for a very large model | A cloud model for those cases only |
What Matricxon and pAIring do
Matricxon runs on both CPUs and GPUs. On NVIDIA cards (MATRICXON_DEVICE=cuda, in BETA) weights are either unpacked for speed or kept compressed to save VRAM. On CPUs, a lot of fine-tuning went into making it work well: it multiplies compressed weights directly, keeps chats cached so a prompt is rarely read twice, and runs mixture-of-experts models efficiently. pAIring works with Matricxon, llama.cpp (with a GPU build it uses your GPU) and a growing list of cloud connectors, so you can start on the hardware you have, add a GPU server later, and switch engines with one setting.
The short answer: start with your CPU. Add a GPU when replies get too slow, usually because of long prompts, many users at once or a large model.