A language model is mostly numbers: billions of weights. Stored at full precision, an 8B model takes about 16 GB, more than most laptops have. Quantization shrinks those numbers so the model fits. Dequantization turns them back into usable values when the model runs, and how an engine does that decides a large part of its speed.
Quantization: fewer bits per weight
A quantized file does not store each weight as a 16-bit number. It groups weights into small blocks, often 256 at a time, and stores:
- one shared scale for the block (a normal number),
- small integer codes for each weight, 2 to 8 bits each.
A weight is then roughly scale × code. Some formats add a per-block offset, or a second, smaller scale for every 16 or 32 weights, to follow the real values more closely.
| Format (GGUF) | Bits per weight | 8B model size |
|---|---|---|
| F16 (no quantization) | 16 | about 16 GB |
| Q8_0 | 8.5 | about 8.5 GB |
| Q4_K_M | about 4.8 | about 4.9 GB |
| IQ2_M | about 2.7 | about 2.9 GB |
The “I-quants” (IQ1 to IQ3) go further: instead of plain integers they store an index into a fixed table of good value patterns (a codebook), plus sign bits. That is how they reach 2 bits per weight and still produce readable text.
Dequantization: getting the numbers back
To compute with a weight, the engine has to know its value, so it reverses the packing: read the scale, read the code, multiply. That is dequantization.
Why bother? Because writing each token means reading nearly all the weights once, and memory speed is the limit (see CPU vs GPU). A model that is 4 times smaller is read about 4 times faster, as long as unpacking it doesn’t eat that gain back.
The price is a little accuracy (Q4_K_M is close to the original, 2-bit types lose more) and the extra work of unpacking.
Two ways to do it
Unpack everything at load time. Turn the whole model back into 16- or 32-bit floats once, then use ordinary fast matrix maths. Simple and fast, but the model needs several times its file size in RAM. A 2.6 GB Ministral 3B file needs about 7.5 GB this way.
Compute on the packed weights. Keep the file as it is and unpack each small block only at the moment it is multiplied. RAM use stays close to the file size, but the unpacking happens again for every token, so it has to be very cheap.
How Matricxon does it
Matricxon uses the second way by default (packed weights). The model file is memory-mapped, and native C kernels multiply the packed blocks directly:
- The input is quantized too. The activation vector (the token’s current state) is turned into 8-bit integers, 256 at a time, with one scale each.
- Integer dot products. Packed weights and the 8-bit input are multiplied as small integers with SIMD instructions that handle 16 bytes at once. Floats come back only once per block: weight scale × input scale × integer sum.
- Unpack once, reuse many times. When reading a prompt, each weight row is unpacked once and reused for every prompt token while it is still in the CPU cache.
- Built for your server. The C library compiles on first start with
-march=native, so it uses whatever your CPU supports.
Every one of the 25 quantization types has a plain Python reference that unpacks a block step by step. The fast kernels, and the Numba versions used when no C compiler is available, are tested against it, so a speed-up can’t quietly change a result. If there is RAM to spare, MATRICXON_ENABLE_QUANTIZED_NATIVE_COMPUTE=false switches to unpacking everything at load time. On an NVIDIA GPU (experimental) weights are unpacked into VRAM by default, where the GPU’s float maths is fastest; MATRICXON_GPU_WEIGHT_MODE=packed keeps them compressed to save VRAM.
A recent example: making 2-bit models usable
Until this week, Matricxon’s I-quant kernels unpacked every weight to a float and multiplied one at a time. A 2-bit Llama 3.1 8B took over 3 minutes for its first word on an older laptop CPU (partly because the laptop was also short on RAM), then about 15 seconds per word.
The fix came from looking at the formats again: every I-quant weight is exactly a small integer scale × a small whole number from its codebook. So these types now use the same integer path as the 4-bit types, with nothing approximated. On one layer of that model:
| IQ2_S, one 14336 × 4096 layer | Before | After |
|---|---|---|
| Writing one token | 44.9 ms | 9.6 ms |
| Reading a 16-token prompt | 529 ms | about 57 ms |
End to end, the first word now comes in 10 to 16 seconds and each further word in about 2.
Which quantization should you pick?
- Q4_K_M is the usual best choice: about a third of the original size, close in quality, and fast to unpack.
- Q8_0 if you have the RAM and want quality closest to the original.
- IQ2 and IQ3 to fit a bigger model into little memory. They lose more quality and are slower to unpack on a CPU, so a smaller model in Q4_K_M is often the better trade.
pAIring shows each model’s expected memory use for the active engine before you download it, so you can see what fits your hardware.