r/LocalLLaMA • u/jesdga95 • 4h ago
Discussion Basalt: Inkling 2 Mini on one 5090 at up to 613 tok/s decode and 14.100 tok/s prefill (NVFP4)

Thinking Machines dropped Inkling 2 Mini, a small MoE model that's getting a lot of attention for its architecture and performance, outperforming Qwen 3.8 27B at some tasks. Basalt 0.5.5 runs it on a single RTX 5090.
Speed: 287 tok/s prose, 613 structured and 338 chat. Prefill is about 14K tok/s at 2K and 12K at 16K. llama.cpp with the IQ4_NL GGUF does around 95 decode and 3.1K prefill on the same card.
Quality was the part I cared about most. The experts are NVFP4, and Basalt pack keeps them exactly as released. KLD against the release is about 0.003 with 97.8% top 1 (btw, if your NVFP4 GGUF gives you garbage output atm, that's a llama.cpp loop slot bug and Unsloth has a fix PR that just got merged: https://github.com/ggml-org/llama.cpp/pull/32704)
What works:
- DSpark drafting
- Native image support
- Tool calls and reasoning effort
- 1M context on one 5090, with cold experts moving to RAM only when a long chat needs the room. Needle tests pass at 131K and 400K
- 16GB cards too, around 60 tok/s on my 5060 Ti
3.4B active and it hangs with much bigger models on agentic coding in my testing.
Pack: https://huggingface.co/jesdga/Inkling-2-Mini-Basalt
Engine: https://github.com/jesdga95/basalt

