A small, readable LLM runtime

An inference runtime written from scratch in Python and PyTorch, with optional native C kernels. It runs GGUF models locally and serves them over an HTTP API, with no llama.cpp, ggml or vLLM code.

Read the docs GitHub

Live view of two chats moving through Matricxon's stages, in pAIring's Stats page

Live chat workflow in pAIring, shown with simulated data.

What it does

Quantized-native

Weights stay packed and are multiplied directly, with 25 quantization types.

CPU and GPU kernels

Native C kernels (AVX2, VNNI, ARM) speed up the heavy steps like matmuls.

Prompt cache

Chats keep their own caches, so users do not re-read long histories.

Encrypted disk cache

Evicted caches are saved with AES encryption and restored on return.

Concurrent users

Replies run side by side, with batched decoding where supported.

Tool calling

Structured tool calls for Qwen, Mistral, Llama 3.x and Gemma 4.

Vision models

Image understanding for LLaVA, moondream2 and Qwen 3.5.

Cluster mode

Nodes share models and caches and agree on membership with Paxos (BETA).

Safe memory use

Models that do not fit in RAM are refused, and idle ones unload.

Get it running

git clone https://github.com/upupabit/matricxon.git
cd matricxon
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/uvicorn app.main:app --host 0.0.0.0 --port 8420

More options and configuration are in the documentation.