A language model is mostly numbers: billions of weights. Stored at full precision, an 8B model takes about 16 GB, more than most laptops have. Quantization shrinks those numbers so the model fits. Dequantization turns them back into usable values when the model runs, and how an engine does that decides a large part of its speed.

Quantization: fewer bits per weight

A quantized file does not store each weight as a 16-bit number. It groups weights into small blocks, often 256 at a time, and stores:

• one shared scale for the block (a normal number),
• small integer codes for each weight, 2 to 8 bits each.

A weight is then roughly scale × code. Some formats add a per-block offset, or a second, smaller scale for every 16 or 32 weights, to follow the real values more closely.

The “I-quants” (IQ1 to IQ3) go further: instead of plain integers they store an index into a fixed table of good value patterns (a codebook), plus sign bits. That is how they reach 2 bits per weight and still produce readable text.

Dequantization: getting the numbers back

To compute with a weight, the engine has to know its value, so it reverses the packing: read the scale, read the code, multiply. That is dequantization.

Why bother? Because writing each token means reading nearly all the weights once, and memory speed is the limit (see CPU vs GPU). A model that is 4 times smaller is read about 4 times faster, as long as unpacking it doesn’t eat that gain back.

The price is a little accuracy (Q4_K_M is close to the original, 2-bit types lose more) and the extra work of unpacking.

Two ways to do it

Unpack everything at load time. Turn the whole model back into 16- or 32-bit floats once, then use ordinary fast matrix maths. Simple and fast, but the model needs several times its file size in RAM. A 2.6 GB Ministral 3B file needs about 7.5 GB this way.

Compute on the packed weights. Keep the file as it is and unpack each small block only at the moment it is multiplied. RAM use stays close to the file size, but the unpacking happens again for every token, so it has to be very cheap.

How Matricxon does it

Matricxon uses the second way by default (packed weights). The model file is memory-mapped, and native C kernels multiply the packed blocks directly:

1. The input is quantized too. The activation vector (the token’s current state) is turned into 8-bit integers, 256 at a time, with one scale each.
2. Integer dot products. Packed weights and the 8-bit input are multiplied as small integers with SIMD instructions that handle 16 bytes at once. Floats come back only once per block: weight scale × input scale × integer sum.
3. Unpack once, reuse many times. When reading a prompt, each weight row is unpacked once and reused for every prompt token while it is still in the CPU cache.
4. Built for your server. The C library compiles on first start with -march=native, so it uses whatever your CPU supports.

Every one of the 25 quantization types has a plain Python reference that unpacks a block step by step. The fast kernels, and the Numba versions used when no C compiler is available, are tested against it, so a speed-up can’t quietly change a result. If there is RAM to spare, MATRICXON_ENABLE_QUANTIZED_NATIVE_COMPUTE=false switches to unpacking everything at load time. On an NVIDIA GPU (experimental) weights are unpacked into VRAM by default, where the GPU’s float maths is fastest; MATRICXON_GPU_WEIGHT_MODE=packed keeps them compressed to save VRAM.

A recent example: making 2-bit models usable

Until this week, Matricxon’s I-quant kernels unpacked every weight to a float and multiplied one at a time. A 2-bit Llama 3.1 8B took over 3 minutes for its first word on an older laptop CPU (partly because the laptop was also short on RAM), then about 15 seconds per word.

The fix came from looking at the formats again: every I-quant weight is exactly a small integer scale × a small whole number from its codebook. So these types now use the same integer path as the 4-bit types, with nothing approximated. On one layer of that model:

End to end, the first word now comes in 10 to 16 seconds and each further word in about 2.

Which quantization should you pick?

• Q4_K_M is the usual best choice: about a third of the original size, close in quality, and fast to unpack.
• Q8_0 if you have the RAM and want quality closest to the original.
• IQ2 and IQ3 to fit a bigger model into little memory. They lose more quality and are slower to unpack on a CPU, so a smaller model in Q4_K_M is often the better trade.

pAIring shows each model’s expected memory use for the active engine before you download it, so you can see what fits your hardware.

Originally published at bytepenny.com.

“You need a GPU for AI” is half true. A graphics card makes some parts of running a language model much faster, and others barely at all. Knowing which is which saves money, and tells you whether the computer you already have is enough.

Both are limited by memory, not maths

When a model writes a reply, every new token means reading nearly all of its weights once. So the speed of writing is set mostly by how fast the chip can read its memory:

That is why a GPU often writes 5 to 15 times faster than a desktop CPU with the same model. It is not that it thinks harder; it reads faster.

The catch: the model has to fit

A GPU is only fast while the whole model, plus the chat’s memory (the KV cache), fits in its own memory (VRAM). Consumer cards have 8 to 24 GB. A 4-bit 8B model (about 5 GB) fits easily. A 4-bit 70B model (about 40 GB) does not fit on any consumer card, but runs on a computer with 64 GB of ordinary RAM.

Splitting a model between the GPU and the CPU works, but helps less than you might expect: every token waits for the slowest part, which is the half that sits in system memory.

Where the GPU really shines

• Reading the prompt. Before the first word of a reply, the model reads your whole message, and with a knowledge base, the documents found for it. This part is pure arithmetic, and a GPU can be 10 to 100 times faster. Long prompts are where a CPU feels slow.
• Many users at once. A GPU can handle several conversations in the same pass over the weights. A CPU mostly takes turns.
• Large models at speed. If you need a 30B to 70B model to answer quickly, you need serious GPU memory, or several cards.

Where the CPU holds its own

• Capacity. 64 GB of RAM costs a fraction of 64 GB of VRAM. Large models fit; they are just slower.
• Cost and what you already own. Every computer has a CPU. For one person, or a small team asking short questions, a well-chosen 3B to 8B model on a modern CPU is often fast enough.
• Mixture-of-experts models. They read only a small part of their weights for each token, so they run far faster on a CPU than their size suggests.
• Simplicity, power and noise. No drivers, no fans at full speed, lower power use, and it runs on any physical or virtual server.

Privacy is the same on both: a model on your own hardware keeps your data on your own hardware, whatever chip runs it.

How to choose

What Matricxon and pAIring do

Matricxon runs on both CPUs and GPUs. On NVIDIA cards (MATRICXON_DEVICE=cuda, in BETA) weights are either unpacked for speed or kept compressed to save VRAM. On CPUs, a lot of fine-tuning went into making it work well: it multiplies compressed weights directly, keeps chats cached so a prompt is rarely read twice, and runs mixture-of-experts models efficiently. pAIring works with Matricxon, llama.cpp (with a GPU build it uses your GPU) and a growing list of cloud connectors, so you can start on the hardware you have, add a GPU server later, and switch engines with one setting.

The short answer: start with your CPU. Add a GPU when replies get too slow, usually because of long prompts, many users at once or a large model.

Originally published at bytepenny.com.

On its own, a language model can only write text. It can’t look up today’s orders, search your wiki or create a ticket. To do that, it needs tools, and someone has to connect each tool to each AI app. MCP (the Model Context Protocol) is an open standard that makes this connection the same everywhere. Anthropic introduced it in late 2024, and it is now supported by many AI apps and tools.

A common way to picture it: MCP is like a USB-C port for AI. Build a tool once as an MCP server, and any app that speaks MCP can use it.

The problem it solves

Without a standard, every app needs its own code for every tool. Ten apps and ten tools means up to a hundred separate integrations, each built and kept up to date on its own. With MCP, each app implements the protocol once and each tool implements it once. Ten plus ten instead of ten times ten.

The three parts

• Host: the AI app people use, such as a chat app or a code editor.
• Client: the connection inside the host that talks to one server.
• Server: a small program that offers capabilities to the model. It can run on the same server or anywhere on the network.

A server can offer three kinds of things:

Messages use JSON-RPC, a simple JSON request and response format. A local server usually talks over standard input and output; a remote one over HTTP (the “streamable HTTP” transport).

How a tool call works

1. Connect and list. The app connects to the server and asks which tools it has. Each tool comes with a name, a plain-language description and a schema for its inputs.
2. Offer. The app passes those tool descriptions to the model with the user’s message.
3. Decide. If a tool would help, the model replies with a tool call instead of text, for example search_tickets with {"status": "open"}.
4. Run. The app sends the call to the server, which does the work and returns the result.
5. Answer. The result goes back to the model, which uses it to write the reply, or calls another tool first.

The model never touches your systems directly. The app stays in the middle and decides what runs.

Using MCP safely

• The model needs tool support. Not every model or engine can call tools. Choose one trained for it.
• Connect only servers you trust. Whatever a tool returns is read by the model, so a dishonest server or a poisoned web page can try to steer it (this is called prompt injection).
• Give the least access needed. A read-only key is safer than an admin one, and keys should be stored encrypted.
• Keep people in control. Let users choose when tools are on, and be careful with tools that change or delete things.

MCP in pAIring

pAIring is an MCP host. An admin adds MCP servers under Admin settings > MCP servers (streamable HTTP), tests the connection, and stores any API key encrypted. A switch in Admin settings > System turns MCP on or off for everyone. Users then turn tools on per chat. Tools work with Matricxon and llama.cpp, so a private model on your own hardware can look things up and take actions while your data stays with you.

Originally published at bytepenny.com.

A language model only knows what it saw in training. It has never read your documents, your product manuals or last week’s meeting notes. RAG (retrieval-augmented generation) fixes that without retraining anything: before the model answers, the app finds the most relevant passages in your documents and hands them to the model together with the question.

It is the most practical way to make a private model useful at work. It also asks a lot from a CPU. Here is why.

How RAG works, step by step

RAG has two phases: preparing your documents once, and then using them for every question.

Preparing the documents (once per file)

1. Extract the text. PDFs, Word files and web pages are turned into plain text.
2. Split it into chunks. Each document is cut into short passages, a few hundred characters each, with a small overlap so that an idea split across a boundary stays whole somewhere.
3. Embed each chunk. An embedding model turns every chunk into a vector, a list of numbers that captures its meaning. Passages about similar topics get similar vectors, even when they use different words.
4. Store the vectors in an index, ready to search.

Answering a question (every time)

5. Embed the question with the same embedding model.
6. Retrieve. Find the chunks whose vectors are closest to the question’s vector, usually the top 3 to 5.
7. Augment the prompt. Put those chunks into the prompt: “Using these passages, answer the question…”
8. Generate. The language model reads everything and writes the answer, ideally quoting where it came from.

The model does not memorize your documents. It reads the few passages it needs, right when it needs them, so new or changed files count at once.

Reranking: a second, closer look

Vector search is fast, but rough. It compares two lists of numbers that were made separately, one for the question and one for each passage, so it finds passages about the same topic, not always the ones that answer the question. Ask “How long is the warranty on the X200?” and a passage about the X100’s warranty may score just as high as the right one.

A reranker fixes that with one extra step between retrieving and augmenting:

1. Shortlist. Vector search picks a wider set, say the top 20 passages instead of 4.
2. Rerank. A reranker model reads the question together with each passage and gives it a relevance score from 0 to 1. Reading both at once lets it notice details, like the model number, that vector search can miss.
3. Keep the best. Only the top few, by the new score, go into the prompt.

The result is better answers without a longer prompt. A reranker is a small model that scores instead of writing, but it still reads every shortlisted passage, so it adds a short wait before the answer starts. On a CPU that is usually worth it: a few seconds spent picking the right 4 passages beats a minute spent reading 10 mediocre ones.

Why RAG is hard on a CPU

When a model runs, there are two kinds of work. Writing the reply is limited by memory speed, and a modern CPU handles it well. Reading the prompt is pure arithmetic, and that is where a CPU is far behind a GPU. RAG makes the reading part much bigger.

• The prompt gets long. A short question is maybe 30 tokens. Add five retrieved chunks and it can grow to a few thousand. A desktop CPU may read tens to a couple of hundred tokens per second with a mid-size model, so the answer can take many seconds to start, even though it then streams quickly.
• It happens on every question. The retrieved passages change with each question, so the work can’t simply be reused. Each new question is read almost from scratch.
• Embedding a whole library takes time. Every chunk of every file goes through the embedding model. For thousands of pages, that is a long first run on a CPU, and it starts again if you change the embedding model.
• Longer prompts use more memory. The model keeps a cache for everything it has read (the KV cache). It grows with each extra chunk, takes RAM, and has to be read again for every word the model writes.

What helps on a CPU

• Retrieve fewer, better chunks. Three good passages beat ten average ones, and every chunk you leave out saves reading time. A reranker is the best way to pick them.
• Keep chunks short and focused. Smaller passages mean less to read and more precise matches.
• Use a small embedding model. Embedding models are tiny compared to chat models, and a small one is usually good enough.
• Embed in the background. Do it once when files are added, not while someone waits for an answer.
• Pick a model that reads fast. A 3B to 8B model at 4-bit, or a mixture-of-experts model, keeps the wait reasonable.
• Reuse what you can. Follow-up questions in the same chat should not read the whole conversation again.

How pAIring and Matricxon handle it

pAIring has a built-in knowledge base. Each user can add private documents, which are split, embedded in the background and searched on every question. By default it retrieves 4 short passages, and each person can change that number in Settings > Knowledge. With Matricxon as the engine, an admin can add a reranker in one click (Settings > Models > Reranker models, Qwen3 Reranker 0.6B): pAIring then shortlists 20 passages and the reranker keeps the best ones. Matricxon checks its reranker scores against llama.cpp’s own. It also keeps a cache for each conversation (and can save it to disk), so a follow-up question reads only what is new, not the whole chat again.

RAG is the simplest way to give a private model real knowledge of your work. On a CPU it works well too, as long as you keep the retrieved context lean.

Originally published at bytepenny.com.

Ask a language model for 2¹⁰⁰, the integral of x² sin x, and it will answer confidently. Sometimes it is right. A language model doesn’t calculate; it predicts the next piece of text, and digits are just text to it. That works for words, but numbers need to be exact.

Wolfram|Alpha was built to do the opposite: compute answers instead of predicting them. Thanks to MCP (the Model Context Protocol), any AI app that supports MCP can now hand math and factual questions to Wolfram|Alpha and get back a computed result. This post explains what that is, what you gain, and where the limits are.

What Wolfram|Alpha is

Wolfram|Alpha is a computational knowledge engine from Wolfram Research, online since 2009. It is not a search engine: instead of finding pages that might contain the answer, it works the answer out. Underneath are two things:

• The Wolfram Language (the language behind Mathematica), with decades of algorithms for algebra, calculus, statistics, number theory, geometry and much more.
• The Wolfram Knowledgebase, a large, curated collection of data on chemistry, physics, astronomy, geography, finance, units and other fields.

It takes plain-language questions such as “solve x^3 - 2x + 1 = 0” or “distance from Earth to Mars now” and returns structured, computed results.

Wolfram|Alpha as an MCP server

An MCP server offers tools that a model can call. In February 2026, Wolfram made its technology available as a “foundation tool” for AI systems, including an official MCP service. Wolfram calls the idea computation-augmented generation (CAG): where RAG adds retrieved text to the prompt, CAG adds computed results.

The Wolfram Cloud MCP server lives at https://agenttools.wolfram.com/mcp. It uses the standard streamable HTTP transport, needs no API key, and is free for limited personal use. It offers three tools:

All three are marked read-only. The cloud service is stateless: each call stands alone, with no saved variables between calls and no file uploads.

There are two other routes. Wolfram Local MCP runs on your own server with Mathematica, Wolfram|One or a licensed Wolfram Engine, and adds persistent sessions, local files and notebook tools. And several community MCP servers wrap the Wolfram|Alpha LLM API, which needs a free App ID from the Wolfram developer portal (a free non-commercial plan is available).

What a tool call looks like

A user asks: “What is the integral of x² sin(x), and what is it from 0 to π?”

1. The model sees the WolframAlpha tool and decides to use it.
2. It calls the tool with the query integrate x^2 sin(x) dx.
3. Wolfram|Alpha computes and returns the answer (this is the actual response):

# Indefinite integral
∫ x^2 sin(x) dx = 2 x sin(x) - (x^2 - 2) cos(x) + constant

# Definite integral
∫_0^π x^2 sin(x) dx = π^2 - 4 ≈ 5.8696

4. The model writes the reply around this exact result, and can add a link to the full Wolfram|Alpha page so the user can check it.

The model does the talking; Wolfram|Alpha does the math.

The math advantages

Language models break numbers into tokens, chunks of characters, and predict them one at a time. Long multiplication, carrying digits and keeping many steps exact are exactly where that fails. Handing the math to Wolfram|Alpha gives you:

• Exact arithmetic. Big integers, fractions and high-precision decimals, computed rather than estimated. 2¹⁰⁰ is 1267650600228229401496703205376, every digit correct.
• Symbolic math. Simplify, factor, expand, differentiate and integrate expressions, with exact forms like π² − 4 instead of a rounded decimal.
• Equation solving. Polynomial, linear and differential equations and systems of equations, including complex roots and exact radicals.
• Units and constants. Unit-aware calculations and conversions, plus physical constants such as the speed of light, without the model typing in values from memory.
• Statistics and number theory. Means, regressions, distributions, primes, factorizations, modular arithmetic.
• Plots. Graphs of functions and data, returned as images the app can show.
• Results you can check. Every answer can come with a link to the full Wolfram|Alpha result, so a person can verify it.

Beyond math

• Curated, up-to-date data. Chemical properties, planet positions, country statistics and exchange rates come from Wolfram’s knowledgebase, not from what the model happened to read during training.
• Fewer hallucinated facts. When a number comes from a tool, the model has no need to invent one.
• Small models with strong math. A compact model running on a CPU will never compute as well as Wolfram|Alpha, and it doesn’t have to. It only needs to recognize when to call the tool. This makes a small private model far more capable without bigger hardware. For more on that trade-off, see running LLMs on CPUs.

Limits and things to know

• The question leaves your network. The query text goes to Wolfram’s cloud. Keep confidential details out of math queries, or use Wolfram Local MCP.
• Internet access is required for the cloud service, and it is meant for small-scale use. For heavy or commercial use, look at Wolfram’s paid APIs.
• The model must support tool calling, and has to choose to call the tool. A model that answers math on its own will still guess.
• English queries work best. The model should translate the question for the tool and answer in the user’s language.
• Code tools need care. WolframLanguageEvaluator runs code in Wolfram’s cloud. It is read-only for your systems, but some apps let you require approval for it.

Using it in pAIring

pAIring is an MCP host, so connecting Wolfram|Alpha takes a minute:

1. Open Admin settings > MCP servers and add a server named Wolfram with the URL https://agenttools.wolfram.com/mcp. No API key or headers are needed.
2. Raise the call timeout to about 60 seconds, since Wolfram Language evaluations can take that long.
3. Test the connection, then make sure MCP is on under Admin settings > System.
4. In a chat, turn tools on and ask a math question.

With a tool-capable model on Matricxon or llama.cpp, your chats stay on your own hardware, and only the math and fact queries go out to Wolfram.

Originally published at bytepenny.com.

You do not need a graphics card to run a useful language model. A modern laptop or desktop CPU can chat, summarize and write code with a well-chosen model, privately and at no running cost. It just rewards a bit of understanding. Here is what we have learned building Matricxon, our CPU-first inference engine.

The one number that matters: memory bandwidth

When a model writes a reply, every new token means reading (nearly) all of the model’s weights from RAM once. The CPU’s arithmetic is rarely the limit; the memory bus is. A useful rule of thumb:

Tokens per second ≈ memory bandwidth ÷ model size in memory

A typical dual-channel DDR4 laptop moves around 40 GB/s, so a 4 GB model tops out near 10 tokens per second, and a 2 GB model near 20. Faster RAM, more memory channels and smaller models all help. Extra cores help much less than you would expect.

Reading your prompt is different: it is compute-bound, so long prompts and big documents take time on a CPU even when replies stream quickly. Keep that in mind when you paste a whole file into a chat.

Pick the right model

• Size first. On a laptop with 8 to 16 GB of RAM, models of 1 to 4 billion parameters feel responsive. A desktop with 32 GB or more handles 7 to 14B well.
• Quantize. GGUF files store weights in compressed blocks. Q4_K_M is the sweet spot for most models: about a quarter of the original size with little quality loss. Small models (under 3B) deserve Q5 or Q6, since every bit counts more for them. Below Q3, quality drops quickly.
• Consider mixture-of-experts. A model such as a 30B with 3B active parameters reads only the experts it uses for each token, so it can run nearly as fast as a 3B model while answering like a much larger one. It still needs RAM for the whole file.

Tune the server

• Threads: physical cores, not all of them. Once memory bandwidth is saturated, more threads add heat, not speed. On laptops, leaving one or two cores free keeps the server responsive and avoids thermal throttling, which can cost more speed than it saves.
• Never swap. A model that spills into swap slows to a crawl. Leave a safe margin of free RAM, and close the browser tab with forty other tabs in it.
• Fill both memory channels. One stick of RAM halves your bandwidth. Two matched sticks are one of the cheapest speed-ups there is.
• Keep the context modest. The chat’s memory (the KV cache) grows with every token of context. A 4K to 8K window is plenty for most chats and keeps RAM free for the model.

What Matricxon does for you

Matricxon is built around these limits, so most of this happens without any setup:

• Packed weights. By default, weights stay compressed in the memory-mapped file and are multiplied directly, with no unpacked copy in RAM. Ministral 3B in Q4_K_M needs about 2.6 GB this way instead of about 7.5 GB. If you have RAM to spare, set MATRICXON_ENABLE_QUANTIZED_NATIVE_COMPUTE=false: unpacked weights are about 2 to 4 times faster per step.
• Native C kernels. All the quantized math runs in C with OpenMP threads, for 25 GGUF quantization types, including the very small IQ types.
• Fewer threads, cooler server. MATRICXON_TORCH_THREADS caps the thread count, for laptops that run hot.
• Prompt caching. Each model keeps several conversations’ caches, and a new message reuses the longest matching prefix, so only the new text is read. Since reading prompts is the slow part on a CPU, this is the biggest win of all.
• Caches that survive. Caches evicted from RAM, or still held when the server stops, are saved to disk with AES encryption. Coming back to a long chat tomorrow does not mean re-reading it.
• Context shift. When a chat outgrows its window, the oldest tokens are dropped from the cache instead of re-reading everything.
• Safe loading. A model that would not fit in free RAM is refused rather than pushed into swap, and idle models unload after a while.
• Several users at once. Replies on one model run side by side, with their steps batched together where the architecture allows.

Quick checklist

1. Choose a model that fits in RAM with room to spare, at Q4_K_M (Q5 or Q6 if small).
2. Try a mixture-of-experts model if you have the RAM but want speed.
3. Use your physical cores minus one or two.
4. Keep chats going instead of starting over, so the prompt cache does its work.
5. Measure: tokens per second tells you at once whether a change helped.

CPU inference will not beat a big GPU, but for private, everyday use it is often fast enough, and it runs on hardware you already own. To try it, start with the Matricxon quick start, or run it with pAIring for a full chat workspace.

Originally published at bytepenny.com.