22 GB model on a 16 GB Mac. 1.42 GB RAM. Coherent 35B reasoning. $0/month.

Runs models that don't fit in RAM on Apple Silicon Macs — no quality compromise, no mmap thrashing.

1. MoE Expert Sniper (the breakthrough): Qwen3.5-35B-A3B at full Q4_K_M quality (22 GB) on a 16 GB Mac with only 1.42 GB of RAM. The router predicts which 8 of 256 experts activate per token — we snipe only those from SSD via F_NOCACHE direct I/O with 8-thread parallel pread. The rest stays on disk. Result: coherent chain-of-thought reasoning at 1.54 tok/s.

> what is 2+2? Thinking Process: 1. Analyze the Request: basic addition (2 + 2). 2. Determine the Answer: 2 + 2 = 4. 3. Self-Correction: The prompt asks for concise. A direct answer is best. 2 + 2 equals 4. 1.54 tok/s · 170 tokens · 1.42 GB RAM 

2. MoE Agent (fast, daily driver): Same Qwen3.5-35B-A3B at IQ2_M (10.6 GB) via llama.cpp. Fits in RAM natively. 30 tok/s with web search, shell commands, and file tools.

3. Dense Flash Streaming (research proof): Qwen3-32B at 4-bit (18.4 GB) streamed from SSD. 4.5 GB pinned, 14.2 GB FFN per token via F_NOCACHE. 0.15 tok/s — 9x faster than mmap thrashing. Produced 387 tokens of coherent chain-of-thought.

4. Tool calling at 2.6 bits per weight. LLM classifies its own intent as plain text — "search" / "shell" / "chat" — and routes itself. 8/8 correct.

5. 64K context via KV cache quantization. Two llama.cpp flags shrink KV cache from 1024 MB to 288 MB. The 9B goes from 32K to 64K context.

brew install llama.cpp pip3 install rich ddgs huggingface-hub --break-system-packages # Download 35B model (10.6 GB) mkdir -p ~/models python3 -c " from huggingface_hub import hf_hub_download hf_hub_download('unsloth/Qwen3.5-35B-A3B-GGUF',  'Qwen3.5-35B-A3B-UD-IQ2_M.gguf', local_dir='$HOME/models/') " # Start server llama-server \ --model ~/models/Qwen3.5-35B-A3B-UD-IQ2_M.gguf \ --port 8000 --host 127.0.0.1 \ --flash-attn on --ctx-size 12288 \ --cache-type-k q4_0 --cache-type-v q4_0 \ --n-gpu-layers 99 --reasoning off -np 1 -t 4 # Run agent python3 agent.py

pip3 install mlx-lm rich ddgs --break-system-packages # Start MLX engine (downloads 9B model on first run) python3 mlx/mlx_engine.py # Run agent python3 agent.py

# Download 9B model python3 -c " from huggingface_hub import hf_hub_download hf_hub_download('unsloth/Qwen3.5-9B-GGUF',  'Qwen3.5-9B-Q4_K_M.gguf', local_dir='$HOME/models/') " # Start server (64K context with quantized KV cache) llama-server \ --model ~/models/Qwen3.5-9B-Q4_K_M.gguf \ --port 8000 --host 127.0.0.1 \ --flash-attn on --ctx-size 65536 \ --cache-type-k q4_0 --cache-type-v q4_0 \ --n-gpu-layers 99 --reasoning off -t 4 # Run agent python3 agent.py

This is the research component. Running a model that genuinely does not fit in RAM at full quality.

Qwen3-32B at MLX 4-bit (group_size=64) is 18.4 GB. On a 16 GB Mac:

• llama.cpp mmap: OS pages the entire model reactively. Result: 0.017 tok/s (system thrashes).
• 2-bit quantization (IQ2_M): Compresses to ~9 GB. Result: 6 tok/s, but quality degrades after ~60 tokens.

Split the model by access pattern:

• Pinned in RAM (4.28 GB): Attention weights, embeddings, norms — always needed, random access
• Streamed from SSD (14.16 GB): FFN layers (~221 MB each) — sequential, loaded per-layer, discarded after use

Key technique: fcntl(F_NOCACHE) + os.pread() bypasses macOS Unified Buffer Cache, preventing page cache pollution. 16KB-aligned custom binary format for DART IOMMU compatibility.

Dense model (Qwen3-32B, 18.4 GB):

MoE model (Qwen3.5-35B-A3B Q4_K_M, 22 GB) — the breakthrough:

The MoE Expert Sniper reads only the 8 active experts per layer from SSD (~14 MB/layer vs 221 MB for dense). The router acts as a built-in prefetch predictor — telling us exactly which weights to load before we need them.

Prompt: "Explain how mixture of experts models work in one paragraph."

<think> Here's a thinking process that leads to the suggested explanation of Mixture of Experts (MoE): 1. Analyze the Request: * Topic: Mixture of Experts (MoE)...

This is the model reasoning about its own architecture — chain-of-thought at 35B quality, streaming from SSD with only 1.42 GB in RAM.

Dense 32B sample (387 tokens, coherent):

"The Flash Attention paper introduces an optimized attention mechanism in transformers, focusing on improving computational and memory efficiency during the self-attention operation... FlashAttention addresses this by reorganizing computation via tiling..."

• MLX 4-bit overhead: group_size=64 scales+biases add 15% overhead. A 32B model at "4-bit" is 18.4 GB, not 16 GB. This pushed it past the RAM limit.
• nn.Module memory leak: Injecting FFN weights via setattr keeps references alive — 3.6 GB growth per 16 layers. Fixed by using mx.quantized_matmul directly on loaded arrays.
• Batching doesn't help: Tested 8-layer batches (16 evals vs 128) — zero speedup. Bottleneck is I/O latency, not GPU sync overhead.
• F_NOCACHE works: 27% speedup over mmap by bypassing the Unified Buffer Cache.
• GGUF is column-major: The correct reshape from dequantized flat data is flat.reshape(ne[1], ne[0]), not flat.reshape(ne[0], ne[1]).T.

See research/flash-streaming/ for full code, audit, and documentation.

KV cache quantization uses llama.cpp's --cache-type-k q4_0 --cache-type-v q4_0 flags, applying per-group 4-bit quantization to key/value states. Inspired by Google's TurboQuant research on extreme KV compression.

The LLM classifies its own intent:

"find me videos on my desktop" → LLM says "shell" → generates find command → executes "who do the lakers play next" → LLM says "search" → rewrites query → DuckDuckGo → answers "explain quantum computing" → LLM says "chat" → streams directly 

Three paths. No hardcoded rules. Upgrading the model upgrades every capability.

The MLX backend adds features llama.cpp can't do:

# Save context after analyzing a codebase curl -X POST localhost:8000/v1/context/save \ -d '{"name":"my-project","prompt":"your codebase here"}' # Next day — resume instantly (0.0003s vs minutes reprocessing) curl -X POST localhost:8000/v1/context/load \ -d '{"name":"my-project"}' # Different Mac — download from R2 (1.5s) curl -X POST localhost:8000/v1/context/download \ -d '{"name":"my-project"}'

See mlx/PROJECT.md for the full research roadmap.

Type / to see all commands:

┌──────────────────────────────────────────────────┐ │ agent.py — LLM-as-Router │ │ search / shell / chat │ ├──────────┬───────────────────────────────────────┤ │ llama.cpp│ MLX backend │ │ backend │ + KV cache save/load │ │ │ + KV cache compression (4-bit) │ │ │ + Cloudflare R2 sync │ │ │ + Flash Streaming (out-of-core) │ ├──────────┴───────────────────────────────────────┤ │ Apple Silicon — Unified Memory + NVMe SSD │ └──────────────────────────────────────────────────┘ 

Same agent.py at every level. Just swap the model.

This project builds on:

• Apple "LLM in a Flash" — Inspiration for out-of-core weight streaming. Our Flash Streaming engine implements the split-model architecture (attention pinned, FFN streamed) with F_NOCACHE direct I/O.
• Google TurboQuant — Inspiration for KV cache compression. We use llama.cpp's per-group 4-bit KV quantization.
• MLX — Apple's native ML framework

• Qwen3.5 — the models
• llama.cpp — inference engine
• Unsloth — GGUF quantizations
• Cloudflare R2 — free object storage
• Rich — terminal UI

MIT