A language model only knows what it saw in training. It has never read your documents, your product manuals or last week’s meeting notes. RAG (retrieval-augmented generation) fixes that without retraining anything: before the model answers, the app finds the most relevant passages in your documents and hands them to the model together with the question.
It is the most practical way to make a private model useful at work. It also asks a lot from a CPU. Here is why.
How RAG works, step by step
RAG has two phases: preparing your documents once, and then using them for every question.
Preparing the documents (once per file)
- Extract the text. PDFs, Word files and web pages are turned into plain text.
- Split it into chunks. Each document is cut into short passages, a few hundred characters each, with a small overlap so that an idea split across a boundary stays whole somewhere.
- Embed each chunk. An embedding model turns every chunk into a vector, a list of numbers that captures its meaning. Passages about similar topics get similar vectors, even when they use different words.
- Store the vectors in an index, ready to search.
Answering a question (every time)
- Embed the question with the same embedding model.
- Retrieve. Find the chunks whose vectors are closest to the question’s vector, usually the top 3 to 5.
- Augment the prompt. Put those chunks into the prompt: “Using these passages, answer the question…”
- Generate. The language model reads everything and writes the answer, ideally quoting where it came from.
The model does not memorize your documents. It reads the few passages it needs, right when it needs them, so new or changed files count at once.
Reranking: a second, closer look
Vector search is fast, but rough. It compares two lists of numbers that were made separately, one for the question and one for each passage, so it finds passages about the same topic, not always the ones that answer the question. Ask “How long is the warranty on the X200?” and a passage about the X100’s warranty may score just as high as the right one.
A reranker fixes that with one extra step between retrieving and augmenting:
- Shortlist. Vector search picks a wider set, say the top 20 passages instead of 4.
- Rerank. A reranker model reads the question together with each passage and gives it a relevance score from 0 to 1. Reading both at once lets it notice details, like the model number, that vector search can miss.
- Keep the best. Only the top few, by the new score, go into the prompt.
The result is better answers without a longer prompt. A reranker is a small model that scores instead of writing, but it still reads every shortlisted passage, so it adds a short wait before the answer starts. On a CPU that is usually worth it: a few seconds spent picking the right 4 passages beats a minute spent reading 10 mediocre ones.
Why RAG is hard on a CPU
When a model runs, there are two kinds of work. Writing the reply is limited by memory speed, and a modern CPU handles it well. Reading the prompt is pure arithmetic, and that is where a CPU is far behind a GPU. RAG makes the reading part much bigger.
- The prompt gets long. A short question is maybe 30 tokens. Add five retrieved chunks and it can grow to a few thousand. A desktop CPU may read tens to a couple of hundred tokens per second with a mid-size model, so the answer can take many seconds to start, even though it then streams quickly.
- It happens on every question. The retrieved passages change with each question, so the work can’t simply be reused. Each new question is read almost from scratch.
- Embedding a whole library takes time. Every chunk of every file goes through the embedding model. For thousands of pages, that is a long first run on a CPU, and it starts again if you change the embedding model.
- Longer prompts use more memory. The model keeps a cache for everything it has read (the KV cache). It grows with each extra chunk, takes RAM, and has to be read again for every word the model writes.
What helps on a CPU
- Retrieve fewer, better chunks. Three good passages beat ten average ones, and every chunk you leave out saves reading time. A reranker is the best way to pick them.
- Keep chunks short and focused. Smaller passages mean less to read and more precise matches.
- Use a small embedding model. Embedding models are tiny compared to chat models, and a small one is usually good enough.
- Embed in the background. Do it once when files are added, not while someone waits for an answer.
- Pick a model that reads fast. A 3B to 8B model at 4-bit, or a mixture-of-experts model, keeps the wait reasonable.
- Reuse what you can. Follow-up questions in the same chat should not read the whole conversation again.
How pAIring and Matricxon handle it
pAIring has a built-in knowledge base. Each user can add private documents, which are split, embedded in the background and searched on every question. By default it retrieves 4 short passages, and each person can change that number in Settings > Knowledge. With Matricxon as the engine, an admin can add a reranker in one click (Settings > Models > Reranker models, Qwen3 Reranker 0.6B): pAIring then shortlists 20 passages and the reranker keeps the best ones. Matricxon checks its reranker scores against llama.cpp’s own. It also keeps a cache for each conversation (and can save it to disk), so a follow-up question reads only what is new, not the whole chat again.
RAG is the simplest way to give a private model real knowledge of your work. On a CPU it works well too, as long as you keep the retrieved context lean.