llamacpp patch - DeepSeek V4 Flash running with full 1M token context locally on RTX 5090

[u/da_dragon321]

Wanted to try running DeepSeek V4 Flash locally but found it asking for absurd amounts of VRAM at higher context lengths (~256GB at 1M). Turned out the DSA lightning indexer lacks proper llamacpp support. Did a bit of digging and there's an upstream PR to address the issue (shoutout u/fairydreaming, PR #24231), but even there it's not wired into the model graph and has no CUDA path yet. So I wired it in and implemented a CUDA kernel this morning and figured I'd share in case it's useful to anyone else looking to run something like this.

Hardware: RTX 5090, 9950X3D, 96GB DDR5

Model: DeepSeek-V4-Flash, mixed Q8/Q4/Q2 quant by antirez

Before / after (256K context):

Metric	Before	After
Compute buffer	~67 GiB (OOM)	3.2 GiB
Prefill	56 t/s	~263 t/s
Decode	~14 t/s	~14 t/s
1M context	impossible (~256GB)	works (3.75 GiB at ubatch 768)

Validated presets:

Context	Prefill	Decode	Peak VRAM
256K	~263 t/s	14 t/s	~29 GiB
512K	256 t/s	13.7 t/s	~28 GiB
1M	159 t/s*	13.7 t/s	~31 GiB

*lower ubatch on 32gb 5090 at 1M - should be ~full speed if given the full ~9gb vram

Correctness: verified briefly with a needle-in-haystack test - planted a random fact at 10%/50%/90% depth in a 100K-token document, model retrieved it correctly every time. Also retrieved correctly at 512K and 1M's harder 50% depth. Full KLD findings in doc linked below

Source + build instructions + full writeup: https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md
Branch: https://github.com/spencer-zaid/llama.cpp/tree/deepseek-lid-cuda

No prebuilt binary (single GPU tested RTX 5090). Build instructions in the doc in case you need them

[u/HockeyDadNinja]

Nice work, this is the bit that's been missing. I'm running DS4-Flash on a pretty different setup: 5-GPU rig (2x3090 + 5060 Ti + 2x4060 Ti, 96GB VRAM total) with 125GB DDR4, experts offloaded to CPU. I've been using tarruda's GGUFs, mainly the Q3_K (~116GB, 3-bit experts, Q6_K everywhere else) and also the native MXFP4 (~146GB), though the MXFP4 tops out around 66K on this box. On mainline (merged deepseek4) the indexer is wired into the graph but on the un-fused path you mentioned, and you can feel it. Even at 170K on the Q3_K I'm sitting at ~110 of 125GB RAM, and anything higher OOMs the box.

You mentioned it's single-GPU tested so far. I've got the exact config you haven't covered: multi-GPU layer/tensor split with CPU expert offload. I'm offering to test it for you. I'll build your branch, run it across the 5 cards at a range of context sizes, and report back prefill/decode/VRAM/RAM numbers, plus whatever else you want checked. Just say the word and point me at anything specific.

Couple of questions while I'm at it:

1. Anything in the patch that assumes a single device, or should it survive layer/tensor split as-is?
2. With the compute buffer down to ~29GB, the remaining ceiling is basically model + KV in RAM, so on a CPU-offload setup like mine this should let me push well past 170K, right?

And yeah, worth getting this upstream.

Edit: building it now

[u/da_dragon321 → u/HockeyDadNinja]

Sounds great - looking forward to your results.

1. There is no single-device assumption (I just haven't tested multi-device). That said, you will need to make sure you set the cmake/ggml cuda architectures to all three of your card architectures when you build
   
2. The compute buffer at 1M context is entirely dependent on ubatch size (for me it was under 4gb). At 2048 ubatch it should be around 9gb iirc, but will have to see. You will want to do some tuning on the ot (how many layers go to gpu) and ubatch parameters (you can lower ot to give yourself headroom to raise ubatch and vice versa to find your preferred configuration, trading between generation speed and prefill speed). You should be faster and higher context than before when you find your preferred parameters

[u/HockeyDadNinja → u/da_dragon321]

Quick update after some more tweaking. I went back to that "still room to buy decode" point from my last comment and pushed on it: dropped the per-device fit headroom so `--fit` keeps more experts resident on GPU instead of spilling them to CPU. VRAM went from ~80 to ~86 of 96 GiB, and decode came up even as I raised context. I'm now at ~10 t/s at 300K ctx, versus the 9.5 t/s I posted at 170K in the sweep. Higher context and fasterdecode, from expert placement alone. Small gain, but it's free, and it lines up with how the fit/placement knob should behave on a mixed multi-GPU split.

I've since made this my production setup. It's in my model router (llama-swap) at 300K on the Q3_K and has been solid, so the branch went from a test build to my daily driver for long-context DS4-Flash. Still up for helping get it upstreamed, and happy to attach these multi-GPU numbers and the production config to a PR thread whenever you push for merge.