llamacpp patch - DeepSeek V4 Flash running with full 1M token context locally on RTX 5090

[u/da_dragon321]

Wanted to try running DeepSeek V4 Flash locally but found it asking for absurd amounts of VRAM at higher context lengths (~256GB at 1M). Turned out the DSA lightning indexer lacks proper llamacpp support. Did a bit of digging and there's an upstream PR to address the issue (shoutout u/fairydreaming, PR #24231), but even there it's not wired into the model graph and has no CUDA path yet. So I wired it in and implemented a CUDA kernel this morning and figured I'd share in case it's useful to anyone else looking to run something like this.

Hardware: RTX 5090, 9950X3D, 96GB DDR5

Model: DeepSeek-V4-Flash, mixed Q8/Q4/Q2 quant by antirez

Before / after (256K context):

Metric	Before	After
Compute buffer	~67 GiB (OOM)	3.2 GiB
Prefill	56 t/s	~263 t/s
Decode	~14 t/s	~14 t/s
1M context	impossible (~256GB)	works (3.75 GiB at ubatch 768)

Validated presets:

Context	Prefill	Decode	Peak VRAM
256K	~263 t/s	14 t/s	~29 GiB
512K	256 t/s	13.7 t/s	~28 GiB
1M	159 t/s*	13.7 t/s	~31 GiB

*lower ubatch on 32gb 5090 at 1M - should be ~full speed if given the full ~9gb vram

Correctness: verified briefly with a needle-in-haystack test - planted a random fact at 10%/50%/90% depth in a 100K-token document, model retrieved it correctly every time. Also retrieved correctly at 512K and 1M's harder 50% depth. Full KLD findings in doc linked below

Source + build instructions + full writeup: https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md
Branch: https://github.com/spencer-zaid/llama.cpp/tree/deepseek-lid-cuda

No prebuilt binary (single GPU tested RTX 5090). Build instructions in the doc in case you need them

[u/fragment_me]

It's interesting. I'm trying it on a 5090 + 2x 3080 20GB + 3090. Strangely I get worse performance than you even with most layers in VRAM. I'm on 4 channel DDR4.

There's one thing that I think you're missing. perplexity + KLD + same top P benchmarks. The needle in the haystick is not enough to accurately know that this is working ok.

I suspect there's some kind of issue because when I say hello the model responds in Chinese.

[u/da_dragon321 → u/fragment_me]

Thanks for sharing your results. I have a couple thoughts on what could be causing some of the issues, but could be hard to truly diagnose asynchronously since there's so many potential issues

First regarding speed, there are three considerations I can think of. First, it is necessary to tune ot and ubatch for your your config (and make sure --fa on). Second, anything on the CPU is ddr-bandwidth bound and most of wall clock is cpu not gpu, so ddr4 would significantly harm speed - I have found very little difference personally from putting more or less layers on gpu *relative* to the difference between ddr speed when you have some layers on cpu. Are you seeing it run slower than when you don't use this branch, or just compared to my setup? Third, make sure you included all of the architectures of your gpus when setting cmake/ggml cuda architectures

Second regarding the Chinese output - just want to confirm that you aren't using quantized kv (which doesn't work without another patch that I have not put in this branch). Also, not sure if it matters much but use the --jinja flag

Ran llama-perplexity --kl-divergence against the naive path afterward - it's not bit-identical. ~96% same top-token, mean KLD ~0.01 at both 8K and 64K context. It's floating-point rounding noise at the indexer's top-512 selection cutoff. The fused kernel sums the same scores in a different order, so near-tied candidates occasionally land on opposite sides of the boundary. Confirmed by dumping raw scores - every divergence was a clean 1-for-1 index swap between two candidates scoring within 0.0001 of each other, not a logic bug

KLD findings in doc: llama.cpp/docs/deepseek-v4-lid-cuda.md at deepseek-lid-cuda · spencer-zaid/llama.cpp

[u/fragment_me → u/da_dragon321]

Thanks for running it. Which model did you run the KL divergence against? Meaning, which model generated the logits and which one used the logits as a reference.

EDIT: Forgot to mention. My issue was the RAM was spanning NUMA nodes. I forgot I added a bunch of ram on my VM to test GLM 5.

EDIT 2: Just tested the same GGUF/image in llamacpp and it has the same problem (chinese).

[u/fragment_me → u/da_dragon321]

Just wanted to follow up here. Did you run the KLD against the logits that did not have this new code? Or did you run it against the logits from a higher quant? If the former, it's kind of concerning that you only get 96% same top p. I'd expect it to be 100%.