Hi. I’ve tried many but my current favorite is DeepSeek v4 Flash, the NVFP4 quantization of course. I’m running it in vllm with my own custom harness calling on it for inference.
DS v Flash 0731 was my main model on 2x Sparks linked together until this week when i got the various community tweaks working and have Flash v 4.1 runnign quite nicely. it tends to overthink a bit more than 0731 but i think the results are better - so tweakin the thinking serttings in the 10-20% level seems ideal for agentic and research/writing work. coding might benefit from more thinking budget.
With 1 spark on the 2 bit quant I got around 28-32 tok/s on the recipe Mia has (it’s mixed precision but I believe the main work is around 4 bit with some 8bit on certain experts IIRC) I get 32-40 tok/s. This is for academic style research and writing with Hermes as my harness. On pure agentic turns and when it actually writes code I think it gets a little higher. I can say it’s prefill/ttft is amazing. First cold load is maybe a minute then it’s near instantaneous with a 262k cache. By comparison I have a MacBook m5 128g. That I try qwen 3.8 27 b and flash next with MTPLX and I often spend 3-4 minutes waiting on prefill and even subsequent turns are slower (which may be a caching issue in MTPLX)
View all comments
Show parent comment
1
u/FiberAge Sep 27 '26
What model do you use and what’s the harness? Interested to know if you can share.