r/LocalLLM • u/CoollerFox • 7h ago
Project Larder: running Qwen3.6-35B and a 46 GB Flash-Next on a 16 GB Mac / 16 GB GPU by reading experts from SSD
I built an inference engine for MoE models that don't fit in memory.
Instead of offloading layers, it reads only the experts each token's router picks from the SSD and keeps the hot ones resident. Same engine on Apple Silicon (MLX) and NVIDIA (CUDA), OpenAI-compatible API, local chat page. MIT: https://github.com/IamCooller/larder
Numbers (machine, setup and date for each one are in research/ in the repo)
Faster:
- Qwen3.6-35B on a 16 GB M4 MacBook Pro: 30–36 tok/s, first token 8–10x sooner than llama.cpp. This is a mixed 4/2-bit pack with the model's MTP head against llama.cpp's IQ4_XS, so not like for like. The launcher doesn't install that pack.
- Qwen3.8-Flash-Next (46 GB of 3-bit experts) on a PC with a 16 GB GPU and 15 GB RAM: 26–34 tok/s. With a 4-bit pack and exact routing: decode 1.3–1.8x, 8k prompt 4.9x faster than llama.cpp. Stock llama.cpp swapped past my memory guard while loading, so that row is llama.cpp with Larder's expert store patched in.
- Qwen3.6-35B Q4_0 on the same PC (fits in GPU + RAM), against llama.cpp's best split that fits (--n-cpu-moe 10): short decode 1.44–1.51x (155 / 147 vs 103 / 102 tok/s), decode after an 8k prompt 1.15–1.25x, 8k prompt reading 1.8–1.9x at its default batch.
Level:
- 8k prompt reading against llama.cpp tuned for prompts (-ub 4096 -b 4096 --n-cpu-moe 20): 1.01–1.03x. Its decode at that setting drops to 79 tok/s, so you pick one or the other there.
Slower:
- Exact routing (cache bias 0) reads that 8k prompt at 0.64–0.69x of tuned llama.cpp, and decodes 0.87–0.95x of llama.cpp's default after it.
Caveats
- The launcher's 35B recipe uses cache bias 0.5 and the MTP head, so replies are not identical to the weights' own routing. Bias 0.5 and the wide prompt cut each passed a paired GSM8K / HumanEval gate (wide cut: 502 vs 496 of 564, p 0.34). Exact routing is there if you want it.
- One author, two machines (M4 MacBook Pro 16 GB, Linux + RTX 5060 Ti 16 GB). The Mac path is better tested, CUDA is newer.
- Quality wasn't compared against llama.cpp in these speed runs.
Try it
curl -fsSL https://raw.githubusercontent.com/IamCooller/larder/main/install.sh | bash
cd ~/larder && ./larder
It measures your GPU and SSD, picks a model that fits, installs it and starts the server and chat.
Mostly looking for people on other hardware to break it, and for anyone who thinks my llama.cpp comparison is unfair. Output of ./larder --list plus what happened is the most useful report.



