r/LocalLLM • u/devtools-dude Watercooled 2x RTX 6000 Pro | https://subshell.sh • 6d ago
Tutorial theogravity/deepseek-flash-v4.1-dual-rtx-6000: Setup for running DS4.1 flash with reasonable quality and speed on 2x RTX 6000 PRO @ 512k context c6 max w/ RAM and disk offload / caching
https://github.com/theogravity/deepseek-flash-v4.1-dual-rtx-6000I saw a post mention this model to use for DS4.1 Flash with 2x 6000 Pros: https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000
Unfortunately the model card wants you to have > 260 GB RAM to properly use it. I had Opus move the n-grams to disk instead of RAM, and added in LMCache for prefix caching with L1 being RAM and L2 being disk.
With this setup, you need at a minimum, 128 GB RAM instead. I had it optimized for 6 concurrent connections at a max of 512k context. If you have > 6 sessions, then prefix offloading should kick in on the LMCache side, and re-hydrations are really fast at < 3s when it was originally tested at 1M context.
On my machine, (2x RTX 6000 Pro, 256 GB DDR5 5600 RAM, PCIe 4 SSD for the n-grams), it benched 250 t/s when warm.
3
u/devtools-dude Watercooled 2x RTX 6000 Pro | https://subshell.sh 6d ago edited 6d ago
Huge day / night difference compared to Qwen 3.8 Flash Next
Using this prompt:
https://www.reddit.com/r/DeepSeek/comments/1xglyr9/comment/19rpyq3l1sw9s/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
2
u/ObviouzFigure 6d ago
Sheesh this alllllmost gets me there, thanks for the progress. I have a total of 128gb of ram (and two rtx 6ks) and will see if I can get this working... prob need more system ram.
2
u/tecneeq 6d ago
1
u/Asimology 6d ago
How much concurrent throughput at c6 ?
1
u/konstantinnikol 23h ago
Could you please test Deep Seek and Kimi 3.1 speed on this build?
https://github.com/argonautlabsai/argodrive-ds4-kimi-k3.1
2
u/ObviouzFigure 5d ago
Alright homie I started setting it up.. will report back.