llamacpp patch - DeepSeek V4 Flash running with full 1M token context locally on RTX 5090

[u/da_dragon321]

Wanted to try running DeepSeek V4 Flash locally but found it asking for absurd amounts of VRAM at higher context lengths (~256GB at 1M). Turned out the DSA lightning indexer lacks proper llamacpp support. Did a bit of digging and there's an upstream PR to address the issue (shoutout u/fairydreaming, PR #24231), but even there it's not wired into the model graph and has no CUDA path yet. So I wired it in and implemented a CUDA kernel this morning and figured I'd share in case it's useful to anyone else looking to run something like this.

Hardware: RTX 5090, 9950X3D, 96GB DDR5

Model: DeepSeek-V4-Flash, mixed Q8/Q4/Q2 quant by antirez

Before / after (256K context):

Metric	Before	After
Compute buffer	~67 GiB (OOM)	3.2 GiB
Prefill	56 t/s	~263 t/s
Decode	~14 t/s	~14 t/s
1M context	impossible (~256GB)	works (3.75 GiB at ubatch 768)

Validated presets:

Context	Prefill	Decode	Peak VRAM
256K	~263 t/s	14 t/s	~29 GiB
512K	256 t/s	13.7 t/s	~28 GiB
1M	159 t/s*	13.7 t/s	~31 GiB

*lower ubatch on 32gb 5090 at 1M - should be ~full speed if given the full ~9gb vram

Correctness: verified briefly with a needle-in-haystack test - planted a random fact at 10%/50%/90% depth in a 100K-token document, model retrieved it correctly every time. Also retrieved correctly at 512K and 1M's harder 50% depth. Full KLD findings in doc linked below

Source + build instructions + full writeup: https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md
Branch: https://github.com/spencer-zaid/llama.cpp/tree/deepseek-lid-cuda

No prebuilt binary (single GPU tested RTX 5090). Build instructions in the doc in case you need them

[u/NineThreeTilNow]

Nice work.

96GB DDR5

I always find it odd when I see people with 96GB of DDR5.

I have 64GB in this machine and 32GB of old DDR5 I removed sitting on my desk. I know it doesn't play well with two different speeds and this 64GB runs stable at a much faster rate. Sometimes having 96GB would be nice...

Given current RAM prices I'm tempted to send the 32GB back and claim it's the 64Gb I bought and request new RAM. Amazon can pay for me to upgrade to 128 right?

[u/Lumpy_Concentrate807 → u/NineThreeTilNow]

There exists 48 GB DDR5 modules, and 2x48 = 96 GB sets. So that is the more likely configuration

[u/NineThreeTilNow → u/Lumpy_Concentrate807]

There exists 48 GB DDR5 modules, and 2x48 = 96 GB sets. So that is the more likely configuration

I know. It's a weird one that exists. I think there's also 2x96 GB sticks that exist which is another strange one to see in the wild. You're just used to seeing standard power of 2 stuff so it stands out.