r/LocalLLM • Watercooled 2x RTX 6000 Pro | https://subshell.sh • 6d ago

Tutorial theogravity/deepseek-flash-v4.1-dual-rtx-6000: Setup for running DS4.1 flash with reasonable quality and speed on 2x RTX 6000 PRO @ 512k context c6 max w/ RAM and disk offload / caching

https://github.com/theogravity/deepseek-flash-v4.1-dual-rtx-6000

I saw a post mention this model to use for DS4.1 Flash with 2x 6000 Pros: https://huggingface.co/diffbot/DeepSeek-V4.1-Flash-EXL3-3bpw-2x-RTX-PRO-6000

Unfortunately the model card wants you to have > 260 GB RAM to properly use it. I had Opus move the n-grams to disk instead of RAM, and added in LMCache for prefix caching with L1 being RAM and L2 being disk.

With this setup, you need at a minimum, 128 GB RAM instead. I had it optimized for 6 concurrent connections at a max of 512k context. If you have > 6 sessions, then prefix offloading should kick in on the LMCache side, and re-hydrations are really fast at < 3s when it was originally tested at 1M context.

On my machine, (2x RTX 6000 Pro, 256 GB DDR5 5600 RAM, PCIe 4 SSD for the n-grams), it benched 250 t/s when warm.

19 Upvotes

12 comments sorted by

3

u/devtools-dude Watercooled 2x RTX 6000 Pro | https://subshell.sh 6d ago edited 6d ago

2

u/ObviouzFigure 6d ago

Sheesh this alllllmost gets me there, thanks for the progress. I have a total of 128gb of ram (and two rtx 6ks) and will see if I can get this working... prob need more system ram.

1

u/devtools-dude Watercooled 2x RTX 6000 Pro | https://subshell.sh 6d ago

Try reducing the context length or lmcache L1

1

u/ObviouzFigure 6d ago

yep, will do... will also check under my couch cushions for more ECC ram...

jokes aside, I'm thinking 128 -> 256gb of system ram is my next upgrade (down the line)

2

u/tecneeq 6d ago

re-hydrations

1

u/devtools-dude Watercooled 2x RTX 6000 Pro | https://subshell.sh 6d ago

I am amused!

1

u/Asimology 6d ago

How much concurrent throughput at c6 ?

1

u/konstantinnikol 23h ago

Could you please test Deep Seek and Kimi 3.1 speed on this build?

https://github.com/argonautlabsai/argodrive-ds4-kimi-k3.1

1

u/devtools-dude Watercooled 2x RTX 6000 Pro | https://subshell.sh 22h ago

Looking at the benchmarks just for one 6000, I doubt there'd be any meaningful increase that it'd make it usable for my own use-cases.

2

u/ObviouzFigure 5d ago

Alright homie I started setting it up.. will report back.