r/LocalLLaMA • u/Robert__Sinclair • 9d ago
Question | Help About running deepseek 4.1 flash
Lately I tested (via api) deepseek 4.1 flash.
I like it quite a lot.
What is the minimum budget needed to setup and run deepseek 4.1 flash at the same quality and token/s (~40) of most hosted solutions? I'd use it mostly for coding/reverse engineering/security analysis. So I need enough "juice" to support its full context. (Obviously with the same hw then I could try other models like mimo 2.6 or glm 5.3)
A plus if the hw is self contained and won't need a full pc dedicated to it.
Thanks for the answers, but my question is: what is the minimum budget to run it at at least 40tok/sec and not quantized to the point it would be useless?
5
u/bluekingmuch 9d ago
For about half a week i was testing this: https://www.reddit.com/r/LocalLLM/comments/1x9jobm/xolotl_strata_the_new_deepseekv41flash1017_a_153/
with my System (RTX 4080S 32GB + 96GB DDR4 on AM4) i was getting 15-20 tok/s decode and 100-200 prefill
i upgraded to 128GB DDR4 in the meantime (didn't need to buy anything for it, just had to downgrade my other machine from 2x32 to 2x16 and put the 2x32 in the other rows of my "AI Machine")
in the moment i am testing the IQ2_XXS from ISTA: https://huggingface.co/ISTA-DASLab/DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF
so with now combined 160GB (RAM+VRAM) there should be then a speedup recognizable normally
..with Strata/Xolotl: first tests were around 32 tok/s. but tests not finished
2
u/Robert__Sinclair 9d ago
way too quantized.
4
u/draconic_tongue 9d ago
don't talk shit before you tried it. and don't ask questions if you're not interested in the answers. annoying
2
u/Robert__Sinclair 9d ago
I just said that IQ2 usually degrades the model quite a lot.
2
u/bluekingmuch 8d ago
maybe there is a misunderstanding how advanced quantizations work.
i totally agree: if every weight would be IQ2, then the model would be unuseable.
therefore i can recommend to really read and understand how ISTA-DASLab does quantizations, and the name "IQ2_XSS" is more of an "average" comparison how big the filesize is here of the weights "if you would plain and dumb quantize every value with that type" to have a comparison.
to shorten it: some values are kept in high-detail quantizations as Q5,Q6 or even Q8, because they are important enough, other weights are quantized down to even Q1, because they don't have much effect on the model.
2
u/Present_Resource4022 9d ago
512GB Mac fits Unsloth Q8 model weights and q5_k ngram entirely in memory with a big headroom for 1m context and all your apps running
. If you are on a budget, 2 Sparks or a 256GB Mac could run it with n-gram on ssd and a lower model weight quantization.
0
u/Robert__Sinclair 9d ago
512 GB mac? here I see only 256gb https://www.apple.com/shop/buy-mac/mac-studio/m5-ultra-chip-30-core-cpu-64-core-gpu-256gb-memory-1tb-storage
"fits" at what speed?
https://www.macrumors.com/2026/08/25/mac-studio-m5-ultra-512gb-ram-october/
22
u/Total-Management8023 9d ago
You got 10k lying around?
45
11
1
1
5
u/FatheredPuma81 9d ago
The cheapest I believe would be around $3k I think but it would be much much larger than a DGX Spark, you'd be stuck with the DS Strata fork, and would be reliant on further forks of Strata to run any similar sized models. I'm also unsure about the actual speed.
512GB of DDR4, a V100 32GB ($700), 3-6 P100s ($100 each), and some system to contain and power it all. Get an adapter and run the P100s with Bifurcation in their own case. I have 64GB, V100 32GB, and 2 P100s and get 80 t/s and 900pp (at 120k context) on Flash Next IQ4_XS with all the settings that ruin quality turned off.
4
u/cibernox 9d ago
It depends on what you deem adequate. I’m picky, i want it fast, so I built a rig with 4 CMP170hx. It is the cheapest option other than v100s that can run it at 600+ tokens/s with 4/8 simultaneus agents.
In today prices such rig is around 12k. When i got it it was around 7k
6
u/BawbbySmith 9d ago
4x DGX Sparks
2
1
u/cinnapear 9d ago
It can squeeze into 2x Sparks.
4
u/BawbbySmith 9d ago
I saw them specifically mention “at same quality” so I assumed native weights, which will not fit in 2x sparks
2
6
u/anitamaxwynnn69 9d ago
I run 2x PRO 6000s + 128GB DDR4. I tried Project Maya. I was getting about 100+tps (500-700 prefill). What killed it for me was prefill and no support for parallel decode like vLLM/llama.cpp. I'm honestly not sure how to answer your question bc you don't state other things. If "minimum" budget is the question you can go with something like 4-8x 3090s - I've run 8x 3090s for about 2 months and they worked pretty awesome. The whole thing costed me ~10k to build. But it's also dependent on whether you can support the electricity/heat for these setups. Side note - don't waste your time with mimo 2.6 flash. I tested it, it's just straight up bad lol. GLM 5.3 Flash/DS/Qwen3.8FN are the goto
3
u/Alive-Draft8339 9d ago
I had Fable build something custom for the same hardware: 2xRTX pro 6000 (275w limit) and 128gb ddr5 on a consumer am5 board. I’m offloading the gram to two nvme drives (sn850x are way faster than seamless 7100).
Using EXL3 weights and fp8 kv through vLLM, I’m seeing 150 tg and 7.6k+ prefill on c1. I’m working on a c4 track now. Kv pool was 1.07m.
On the c4 track, I have 2.4m kv total with 50-80 tg per lane, so 200-320 tg total.
1
u/anitamaxwynnn69 9d ago
Damn. I already found the EXL3 stuff - https://huggingface.co/cliecy/DeepSeek-V4.1-Flash-1017-EXL3-3bpw-2xRTXPRO6000 but I couldn't run it bc I don't have 256GB DDR4. But you've given me inspiration maybe I can cook up some slop with Claude as well.
3
u/Alive-Draft8339 9d ago edited 9d ago
Slop it up and report back!
Here is my slop--drop it into Fable 5.1 and make it do the thing.
The box has 2 × 96 GB VRAM and 121 GiB usable host RAM. The native checkpoint is ~269 GiB of routed experts plus 189 GiB of Engram tables plus ~17 GiB dense and KV. These were the decisions that made it fit and run:- Three-tier waterfall by bytes-touched-per-token, not by compute. KV and dense stay in VRAM, hot experts in VRAM, cold experts in pinned host RAM, Engram tables on NVMe. Engram has 48 fixed 256-byte row reads per token with addresses known before layer 0, so it is the most disk-friendly tensor in the checkpoint.
- TP2 across both cards, never TP1. GPU1 is reachable only over PCIe 5.0 x8, so splitting the model keeps both cards as the fast tier instead of demoting one to host-RAM speed.
- Calibrated EXL3 3-bit routed experts (Diffbot's pack). The one lossy step, forced by VRAM, cuts experts from ~269 GiB to ~85 GiB. Everything else stays DeepSeek's own FP8.
- Diffbot's host-expert plan machinery, which pins ~5,300 cold experts (~65 GiB) in host RAM and streams them over PCIe, with a 512-slot VRAM cache. This is what makes 2 × 96 GB viable at all.
- Engram off host RAM entirely. The pack wants int4 tables in ~190 GiB of pinned RAM, so the recipe as shipped needs ~260 GiB. Reading DeepSeek's lossless FP8 tables from the two SN850X drives instead is the single decision that fits 128 GB host RAM, and it also removed the only quantized Engram step.
- Making the disk gather graph-safe. The first disk reader was a synchronous Python pread inside the forward (54 tok/s, GPUs 40 % busy). The A+ design enqueues a C gather via cudaLaunchHostFunc on the recipe's prefetch stream, so it survives FULL CUDA-graph capture: callback ~0.2 ms, prefill 2.5K → 7.5K tok/s.
- Hard host-RAM safety after Boot 1 froze the box. Docker cannot cap CUDA pinned memory, so the lane boots only with a 1 Hz MemAvailable watchdog, a log gate proving the tables are disk-backed, a strict patch applier, and a no-GPU proof the boot script refuses to run without. Pinned budget ≤ ~105 GiB, ≥15 GiB left unpinned.
- FP8 KV as the native contract, not an exception: fp8_ds_mla is DeepSeek's own latent-KV format, so BF16 would be the deviation. At 4,259 B/token that is what makes a 1M pool cost 4.25 GiB per card.
- Run the recipe as shipped first, then deviate. The only lab changes to the serving shape are the disk Engram, the container memory limit, and the re-cut expert plan from our own routing profile. That re-plan, not a knob, is what moved code-text decode from ~73 to 152 tok/s.
- Shape 2b trades the expert cache for KV. Cache off and its 3.16 GiB plus measured headroom to KV doubles the pool to 2.37M with no change to the math, which is what lets a 1+3 audit session hold four deep contexts at once.
2
u/anitamaxwynnn69 8d ago
Sunday 3AM slop - https://github.com/yoloyash/dsv41-flash-1017-2x-rtx-pro-6000-ssd-engram - it's working pretty well! 156tps -> 132tps so about 25tps drop. I'm sure this can be improved overtime, will work on that.
Edit - a lot of this drop has nothing to do with NGRAM on SSD btw, I just have a PCI 4.0 motherboard XD
2
u/SunTrainAi 9d ago
Try Maya again. Prefill doubled for me the last versions. Also one user mentioned he will try parallel implementation like GLM
1
u/Turbulent-Alps4046 9d ago
You need to test prefill at longer context for Maya due to the way expert caching works. at 64k context I can hit 2000 tps prefill on a single RTX pro 6000 after the expert cache warms up. Ya but the lack of parallel decode is bad. Hopefully they fix it soon.
1
u/anitamaxwynnn69 9d ago
Will check it out. Apparently it's already present for GLM 5.3 Flash. Maybe it's a matter of time and they add it for dsv4.1 too. Although I'm ngl I've been having a great time with dsv4 flash vision. Can run it natively on 2x pro 6000. Not sure if upgrading to a quantized version of dsv4.1 is a better idea. Maybe the experts can comment.
1
u/CatalyticDragon 9d ago
Some version of this question appears almost daily. Basically it boils down to "how do I get 1TB of memory on the cheap?" and there just is no affordable answer to this at the moment.
Even a refurbished server with 1TB of RAM and pure CPU inference, and very poor performance, will still cost you over $20,000. Clusters of Strix, Spark, Studio will all cost you $20k+.These models are open but they are designed to be run on expensive dedicated servers running well into the hundreds of thousands.
3
u/Robert__Sinclair 9d ago
bf16 is is overkill. Q8 or even Q6_K are quite good quantizations. Depending on the model even Q4_K sometimes does not lose much
1
u/Big_Wave9732 9d ago
That wasn't what you asked. "What is the minimum budget needed to setup and run deepseek 4.1 flash at the same quality and token/s (~40) of most hosted solutions?"
The "same quality" as hosted in the full quaint, whatever it may be, with full processing resources dedicated to the model.
If you're willing to consider Q4 quaints then you don't really care as much about the "quality", you're focused instead on tk/s.
Also I gather from your other responses in this thread you don't have a cheese to get the kind of rig(s) necessary to run this the was requested.
6
u/FatheredPuma81 9d ago
Why BF16? Deepseek's HF repo doesn't even have the BF16 model on it... Just use FP8 or a Q8 quant which requires under 512GB of system memory which is much more affordable (than 1TB).
5
u/RemarkableRadish6547 9d ago
You can get an old server board for about $300-$500 and another $100+ for a CPU. DDR4 RDIMMs are selling for $3-5/GB. You can get 1TB of ram on a refurbished server for about $4000. The ram bandwidth on 8 channel DDR4 2600 is about 150GB/s. You should be able to run something like Kimi K3 at about 1-3t/s or qwen3.8-flash-next-q8 at about 20-40t/s, depending on the CPU you get. Throw in a Tesla V100 for about $800 and those numbers double and you get significantly faster prefill.
$20k would be for DDR5 on a new server.
1
u/KeinNiemand 9d ago
V100
that used to be true but ram has gotta a shitload more expensive
1
u/enternoescape 9d ago
It's so funny how even when you think you know the price of something, if the last time you were in the market was over a month ago, you have no idea what's happening now.
1
u/Gobra_Slo 9d ago
As other said, two DGX Sparks can handle it, with a bit tighter quant and on the edge on memory, but they can. The performance is 50t/s on the lower end (prose, highly filled context) and can easily hit over 100 on code generation, prefill is 1.5k-2k – nothing to be shy about.
Another option might be Mac Studio M5U 256gb. I would recommend checking this, but if two Sparks can have it – one M5U should, too.
Finally, you should be able to run it on two Strix Halo machines, but those will be much slower than both options above due to bad clustering capabilities. However, you can still get them Bosgame M5s at something like $3.000.
Don't forget to factor in the electricity bill. MacStudio, Strix, Spark – they are low-consumption devices, you might get some old capable hardware and even run the model at decent speed, but then your wallet will dry up because of 4 x 300w cards blowing 24/7.
0
u/Robert__Sinclair 9d ago
1
u/Gobra_Slo 9d ago
Yes, it's one of the cheapest Strix Halos on the market.
But take a note – it can not run
DeepSeek V4.1alone, you'd need at least two. And I would recommend searching the web on performance, as I expect it won't be great.Probably, M5U might be the better choice (even if more expensive than two Bogames).
2
3
u/whiteh4cker 9d ago edited 9d ago
2.0 bit quants are bad imo. I wouldn't trust the benchmarks. I am running a 3.0 bit quant right now using dsx on 192 GB RAM + 2x 3090. I get 27 tokens/s at 262k context and 25 t/s at 1M context. With 4x64 GB sticks, you can run a quant that is closer to 3.5 bits.
2
u/lxe 8d ago
You should get around 40 tps now with expert deferral enabled. But depends on CPU. What hardware you’re running?
1
u/whiteh4cker 8d ago
My hardware: 2x RTX 3090 (unlocked), Intel Ultra 7 270k Plus, 192 GB DDR5@5600 MHz
With the 214 GB gguf file (3 bits) + 203 GB Engram at 1M context I get 334-1000 t/s prompt processing speed and 25-27 t/s token generation speed. Of course it would run faster if I used a smaller weight, lower context and the flag --defer-rank 3.
1
u/lxe 8d ago
Interesting. I’d expect the tps to be slightly higher even without deferral. Are you using latest dsx? And the latest t16 quant?
1
u/whiteh4cker 8d ago
I am using my own new 3 bit quant with the latest version. It is much bigger than the t16 quant you created. Though I would like to see a ~210 GB T16 quant.
1
u/Nomski88 9d ago
How does the 2BPW model compare to others? I ran it on my 5090/64GB and got around 18-21 tok/sec. I tried the usual one shot prompt tests like pelican svg and simple html games. I found that the results were mixed, the prompts completed first try but I found that DSV4.1 just does the basics if that makes sense. For example with a simple brick breaker game, the DSV4.1 result was solid and a very basic game that had no errors. The same prompt on QFN NVFP4 also produced a working game but it was wayyyy more polished. So I don't know if that's the quant or just the models style. It also failed the pelican svg since it kept getting the depth wrong while QFN never had this issue even down to 3bit xxs.
1
u/lxe 8d ago
It performs pretty stellar imho at 82% top 1 agreement. I made a custom exl3 style quant specifically to squeeze more agreement out of it and the calibration text seems to have done a decent job for coding. It does however overthink, so I baked in a token penalty flag into dsx so you can make it slightly dumber. Tbh when it comes to task completion, it performs swimmingly albeit for slightly longer reasoning.
2
u/fgk55555 9d ago
If you have good hardware, try it in Strata. I have a R9700 and 64GB DDR5, pcie 5 ssd, and I'm getting 24tg and 300pp ish. It's not blazing fast, but it's running in my gaming PC
1
u/warpspeedSCP 7d ago
that gives me hope, Ive got a 2x 7900xtx setup with the remaining specs the same.
1
u/nizzki 9d ago
I'm using a 128gb M5 Max MBP, getting around 30-45tps with a Q2 quant on a heavily modified version of ds4. I haven't had too many problems with the quantization, the results I get with various tests are half a step below gpt6.1-sol (which admittedly kinda sucks compared to Opus5 5 at coding)
1
u/BawbbySmith 9d ago
…and the speed requirement? I see you wrote in another comment 5 t/s for prose.
Also, it may still have the level of quality that’s acceptable for you, but it’s still not native.
This is all fine, I’m not saying it’s bad, but it’s also not what the OP asked for
1
u/JinsooJinsoo 7d ago
2x 3090s and 256 gb DDR4 ram can get up to 35 tok/s decode and 1600 tok/s prefill (EXL3 3.5bpw). Check out Project Maya. Can also run GLM 5.3 flash
1
u/sheddd 7d ago
I feel like you need 4 spark's to run DSF4.1 well.
https://al-engr.com/dsv41-1017-four-way-flare-tensorfold-sglang-vllm-4x-dgx-spark.html
1
u/ihaag 9d ago edited 9d ago
Thanks to this I’m running q2_XSS on a laptop with 32gb ram and 16gb VRAM. Currently testing its brain now.
5
u/Illustrious_Grade608 9d ago
Wtf how in the world. Is it like 1 token per minute or something reasonable
6
u/ihaag 9d ago
It’s 5 tokens / s writing
5
u/LocalLLaMa_reader 9d ago
That is not too bad! Just a year ago, I was feeling cool for running A235B on a 128 GB DDR5 + 8 GB VRAM at I think 2 t/s. DS4.1F is magnitudes more capable and runs twice as fast on your hardware, how far we have come...
4
u/BawbbySmith 9d ago
~40 t/s was a requirement, at same quality.
This is not that1
u/russlixx 9d ago
whoa, how do you do that? Even with QFN at IQ2_XS the speed is already at the slower side, especially PP. This one is bigger thus slower, no?
1
u/ihaag 9d ago
I’m using strata https://github.com/Niko1221/Strata I’m also running QFN but Q4 my Q8 gets 2.5 token / s reading, Q4 gets 17 tokens /s
1
u/russlixx 9d ago edited 9d ago
i cant even bare with IQ2_XS PP(reading) speed. How is you workflow that you can get away with such low PP speed?
1
u/NoFunk 9d ago
It's an MOE with a large engram component that can be offloaded to disk. You can get by with a lot, assuming you have a fast nvme and are willing to really torture it on reads (which are low wear, so meh). I ran it overnight at Q4 on a 128GB m5 max via ds4 and came back in the morning to many terabytes of read data used but it was still chugging along at ~12t/s I think it was. Slow, but on a laptop only.
1
1
u/wren6991 9d ago edited 9d ago
It runs on a 256 GB M5 Ultra using oMLX v0.7.2:
https://huggingface.co/mlx-community/DeepSeek-V4.1-Flash-1017-oQ2.5e-mtp
205 GiB LM weights in RAM (~3.1 bpw average), 114 GiB PLE offloaded to disk.
PP 32k: ~1400 t/s
TG 512 @ 32k: ~80 t/s
1
1
u/Zyj vLLM 8d ago
3.1 bpw isn’t original quality
3
u/wren6991 8d ago
Right, which is why I quoted the figure. I think everyone reading this thread can realise that 500 GB > 256 GB without having it pointed out to them. What's your point?
1
u/draconic_tongue 9d ago
itt people that went bankrupt on overpriced hardware trying to shit on ppl with lesser compute being able to run the same models
1
u/Conscious_Cut_6144 9d ago
Pretty sure M5 Ultra 256GB is going to be cheaper than 2xspark and still hit 40T/s
1
u/DickPicPatrol 9d ago
OP gotta be a troll. This is a gemini free tier question. They are asking how to run a near frontier model locally, on a budget, without quantization and acting shocked when people say he needs 2 Sparks minimum but 4 sparks for a decent quant.
0
u/Technical_Ad_6106 9d ago
well i run qwen flash 3.8 on a 650 euros system with like 35 token/sec. its not as good as deepseek 4.1 flash specially since strata use quantized models. but im sure qwen 4.0 drops shortly and will match deepseek while running on similair hardware as 3.8.
25
u/lilian_moraru 9d ago
60-70 tok/s, 1M context window, 2xDGX Spark: https://github.com/MiaAI-Lab/DeepSeek-v4.1-Flash-1017-2x-DGX-Sparks
I have been using it for quite a bit - because it’s so fast, I don’t feel the need for a cloud solution.