colibrì: Run a 744B-Parameter LLM on Your Laptop with Only 25GB RAM
colibrì: Run a 744B-Parameter LLM on Your Laptop with Only 25GB RAM 🐦
What is it? colibrì is a pure C, zero-dependency inference engine that runs GLM-5.2 — a 744-billion parameter Mixture-of-Experts (MoE) model — on a consumer machine with just ~25 GB of RAM and no GPU. It streams model experts from disk on demand, keeping only the dense layers resident in memory.
Why it's trending: colibrì hit 7,300+ GitHub stars in under two weeks because it fundamentally changes what "self-hosted AI" means. Until now, running a frontier-class 744B model required multiple datacenter GPUs (H100s at $30K+ each). colibrì does it on a laptop with an NVMe drive — no GPU, no cloud API, no Python runtime. The tradeoff is speed (0.05–2 tok/s depending on hardware), but for offline research, batch analysis, and privacy-sensitive workloads, it's a breakthrough.
📋 Prerequisites
Before you start, make sure you have:
- A Linux machine (or WSL2 on Windows, or macOS, or Windows 11 native with MinGW-w64)
- ~370 GB of free NVMe/SSD space for the int4 model weights
- ~25 GB of RAM (16 GB bare minimum, 32+ GB recommended for caching)
- gcc with OpenMP and AVX2 support (any modern x86-64 or ARM processor)
- Python 3.10+ (only for the one-time model conversion, not at runtime)
- Patience — colibrì is disk-bound; a cold start runs at ~0.05–0.1 tok/s on modest hardware
🧠 Architecture Overview
colibrì exploits a key property of Mixture-of-Experts models: a 744B MoE activates only ~40B parameters per token. The dense parts (attention, shared experts, embeddings — ~17B params) stay in RAM at int4 (~9.9 GB). The 21,504 routed experts (~370 GB total at int4) live on disk and are streamed on demand with an LRU cache, optional pinned hot-store, and OS page cache as a free L2 tier.
The engine is a single C file (c/glm.c, ~2,400 lines) with small headers. No BLAS, no Python at runtime, no GPU required. Key components:
- MLA Attention — Multi-head Latent Attention with compressed KV-cache (576 floats/token vs 32,768 — 57× smaller)
- Sigmoid Router — DeepSeek-V3-style routing with no auxiliary loss
- Native MTP Speculative Decoding — GLM-5.2's own multi-token-prediction head drafts tokens at 2.2–2.8 tokens/forward (int8 head required)
- Grammar-Forced Drafts — On constrained outputs (JSON, function calling), the grammar itself generates pre-accepted tokens
- Learning Cache — Records which experts you use and auto-pins the hottest ones in spare RAM
- KV-Cache Persistence — Conversations persist across restarts with zero re-prefill
🔧 Setup & Installation
Step 1: Clone and Build
git clone https://github.com/JustVugg/colibri.git
cd colibri/c
./setup.shThe setup script checks for gcc, OpenMP, and AVX2, then builds the engine and runs a self-test. You should see "32/32 positions" confirming the architecture is correct.
Step 2: Download the Pre-Converted Model
The easiest path is to download a pre-converted int4 model with int8 MTP heads (critical for speculative decoding):
# Install git-lfs first
git lfs install
# Download the model (~370 GB)
git clone https://huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp /path/to/glm52_i4⚠️ Critical: The MTP head must be int8. The int4 MTP head gives 0% draft acceptance, meaning speculation silently never engages. Verify with
ls -l /path/to/glm52_i4/out-mtp-*— correct int8 files are ~3.5 GB, ~5.3 GB, and ~1.0 GB.
Step 3: Measure Your Disk Performance
Before running the model, benchmark your NVMe to set expectations:
cd c
gcc -O2 -fopenmp iobench.c -o iobench
./iobench /path/to/glm52_i4/out-00069.safetensors 19 64 8 1 # O_DIRECT modeThis tests the random 19 MB reads that colibrì uses during inference. A PCIe 4.0 NVMe typically gets 3–5 GB/s; PCIe 5.0 reaches 8–12 GB/s.
Step 4: Chat!
COLI_MODEL=/path/to/glm52_i4 ./coli chatOn first launch, the engine allocates its cache (~20 GB RSS peak), loads the dense layers into RAM (~30 seconds), and then you're ready to chat.
🚀 Advanced Configuration
Performance Tuning
colibrì has several knobs to squeeze more speed from your hardware:
| Setting | Description | Recommended |
|---|---|---|
--topp 0.7 |
Adaptive expert top-p sampling (30–40% less disk I/O) | Start here |
--ram 48 |
Set explicit RAM budget (default: auto-detect from MemAvailable) | 48 GB+ if available |
DRAFT=4 |
MTP speculative draft depth (requires int8 head) | 4 (default) |
GRAMMAR=file.gbnf |
Grammar-forced drafts for JSON/structured output | Use for API mode |
THINK=1 |
Enable GLM-5.2's reasoning block | Optional |
PIN_GB=40 |
Pin ~40 GB of hottest experts in RAM after learning usage | After running a session |
# Record expert usage first
STATS=stats.txt ./coli chat --temp 0.7 --topp 0.7
# Then pin learned hot experts
PIN=stats.txt PIN_GB=40 COLI_MODEL=/path/to/glm52_i4 ./coli chat --temp 0.7 --topp 0.7OpenAI-Compatible API
colibrì exposes a standard API endpoint:
COLI_MODEL=/path/to/glm52_i4 COLI_API_KEY=your-key ./coli serve \
--host 127.0.0.1 --port 8000 --model-id glm-5.2-colibri
# Test it
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Authorization: Bearer your-key' \
-H 'Content-Type: application/json' \
-d '{"model": "glm-5.2-colibri", "messages": [{"role": "user", "content": "Hello!"}], "stream": true}'GPU Tier (Optional)
If you have an NVIDIA GPU with 8+ GB VRAM, colibrì can use it as an expert accelerator:
make CUDA=1 # Build with CUDA support
COLI_CUDA=1 COLI_GPU=0 ./coli chat # Enable GPU expert tier✅ Verification Checklist
After setup, verify everything works:
- Engine self-test passes:
./setup.shshows"32/32 positions" - MTP head is int8:
ls -l /path/to/glm52_i4/out-mtp-*shows correct file sizes - Model loads without errors:
COLI_MODEL=/path/ ./coli chatshows"ready in ~30s · resident ~9.9 GB" - Model responds coherently: test with a simple prompt
- API endpoint works (if using serve mode):
curlreturns valid JSON - Expert cache auto-sizes to your available RAM (check RSS in stats line)
📊 Performance Expectations
colibrì's speed depends entirely on your hardware. Here are real measured benchmarks:
| Hardware | Disk Speed | Config | Tokens/s |
|---|---|---|---|
| 12-core WSL2, 25 GB RAM, NVMe VHDX | ~1 GB/s | Default | 0.05–0.1 |
| Apple M5 Max, 128 GB unified | ~4 GB/s | Default, MTP off | 1.06 |
| Apple M5 Max + Metal backend, 128 GB | — | Metal on, 46.9 GB pin | 2.06 |
| Ryzen 9 9950X + PCIe 5.0 NVMe, 123 GB | 8.81 GB/s | Learned pin | 0.28 |
| Ryzen AI Max+ 395, 128 GB, Optane 905p | 3.27 GB/s | Learned pin 47.6 GB | 0.40 |
| EPYC 7443, 430 GB RAM, NVMe RAID | ~1 GB/s | 77.5 GB pin, 98% hit | 1.00 |
| Mac Mini M4 Pro, 48 GB unified, Metal | 6.59 GB/s | Metal on, 38 GB RAM | 0.30 |
Key insight: RAM is the most important factor. More RAM → bigger expert cache → fewer disk reads → faster inference. A machine with 128+ GB RAM and a good NVMe can approach interactive speeds.
📚 Resources
- GitHub Repository: github.com/JustVugg/colibri
- Pre-converted int4 Model (with int8 MTP): huggingface.co/mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp
- Original int4 Model Mirror: huggingface.co/jlnsrk/GLM-5.2-colibri-int4
- GLM-5.2 by Zhipu AI (Z.ai): bigmodel.ai
- License: Apache 2.0 (engine), MIT (model weights)