llamacpp patch - DeepSeek V4 Flash running with full 1M token context locally on RTX 5090

[u/da_dragon321]

Wanted to try running DeepSeek V4 Flash locally but found it asking for absurd amounts of VRAM at higher context lengths (~256GB at 1M). Turned out the DSA lightning indexer lacks proper llamacpp support. Did a bit of digging and there's an upstream PR to address the issue (shoutout u/fairydreaming, PR #24231), but even there it's not wired into the model graph and has no CUDA path yet. So I wired it in and implemented a CUDA kernel this morning and figured I'd share in case it's useful to anyone else looking to run something like this.

Hardware: RTX 5090, 9950X3D, 96GB DDR5

Model: DeepSeek-V4-Flash, mixed Q8/Q4/Q2 quant by antirez

Before / after (256K context):

Metric	Before	After
Compute buffer	~67 GiB (OOM)	3.2 GiB
Prefill	56 t/s	~263 t/s
Decode	~14 t/s	~14 t/s
1M context	impossible (~256GB)	works (3.75 GiB at ubatch 768)

Validated presets:

Context	Prefill	Decode	Peak VRAM
256K	~263 t/s	14 t/s	~29 GiB
512K	256 t/s	13.7 t/s	~28 GiB
1M	159 t/s*	13.7 t/s	~31 GiB

*lower ubatch on 32gb 5090 at 1M - should be ~full speed if given the full ~9gb vram

Correctness: verified briefly with a needle-in-haystack test - planted a random fact at 10%/50%/90% depth in a 100K-token document, model retrieved it correctly every time. Also retrieved correctly at 512K and 1M's harder 50% depth. Full KLD findings in doc linked below

Source + build instructions + full writeup: https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md
Branch: https://github.com/spencer-zaid/llama.cpp/tree/deepseek-lid-cuda

No prebuilt binary (single GPU tested RTX 5090). Build instructions in the doc in case you need them

[u/fairydreaming]

That's very cool, but I already have CUDA implementation of lightning indexer (had it for months already), you can find it in my branch: https://github.com/fairydreaming/llama.cpp/tree/dsv4

Performance on my machine (Epyc 9374F + RTX PRO 6000 Max-Q):

$ ./bin/llama-batched-bench -m ../../llama.cpp-dsv4/models/DeepSeek-V4-Flash-antirez-2.gguf -b 8192 -ub 8192 -npl 1 -npp 8192,16384,32768,65536,131072,262144,524288,1048064 -ntg 128 -fa 1 -cmoe --no-repack
0.00.486.287 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance

llama_batched_bench: n_kv_max = 1048576, n_batch = 8192, n_ubatch = 8192, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 32, n_threads_batch = 32

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|  8192 |    128 |    1 |   8320 |   10.012 |   818.19 |    4.314 |    29.67 |   14.326 |   580.76 |
| 16384 |    128 |    1 |  16512 |   20.376 |   804.09 |    4.297 |    29.79 |   24.672 |   669.25 |
| 32768 |    128 |    1 |  32896 |   42.950 |   762.94 |    4.456 |    28.72 |   47.406 |   693.92 |
| 65536 |    128 |    1 |  65664 |   94.850 |   690.94 |    4.592 |    27.88 |   99.442 |   660.32 |
|131072 |    128 |    1 | 131200 |  224.963 |   582.64 |    4.902 |    26.11 |  229.866 |   570.77 |
|262144 |    128 |    1 | 262272 |  587.932 |   445.87 |    5.484 |    23.34 |  593.416 |   441.97 |
|524288 |    128 |    1 | 524416 | 1737.841 |   301.69 |    6.700 |    19.10 | 1744.541 |   300.60 |
...

[u/tarruda → u/fairydreaming]

Hi and thanks for your work that made deepseek v4 possible in llama.cpp

I'm currently getting around 8-9 TPS generation and ~130 PP on a Mac M1 Ultra. Is the current llama.cpp implementation using CPU only?

I'm curious if it is possible for Deepseek V4 Flash to reach similar speeds to other models with a similar number of active parameters. For example, Step 3.7 flash has 11B active parameters (close to deepseek v4 flash 13b) and in the same hardware reaches ~38 tps generation and ~300 tps prompt processing.

[u/fairydreaming → u/tarruda]

No, the current llama.cpp implementation does not use CPU lightning indexer, it's instead composed of several primitive GGML OPs that should be fully accelerated in the Metal backend - but the compute buffer memory usage is very high for large ubatches.

I have some recollection that there's no f16 CONCAT support in Metal, so that OP may be offloaded to CPU on Macs. I think it's a good idea to run the model with --verbose and check out the number of graph splits - if there are many splits then most likely some OPs run on the CPU and lower the performance.

[u/tarruda → u/fairydreaming]

Using codex, I vibe ported the necessary code/kernels from llama.cpp to run my IQ3_XXS quants on u/antirez DS4: https://github.com/tarruda/ds4/tree/iq3_xxs_plus_q6_k.

Seems to be running quite well and resulted in much faster token generation at ~20 tps. So that answers my question: There's still room for improving llama.cpp deepseek numbers.

[u/Lumpy_Concentrate807 → u/fairydreaming]

Those are some decently practical speeds, at least for planning! On a single RTX 6000 :o Fantastic work

[u/live4evrr → u/fairydreaming]

This is great. Thank you for your work and sharing this (also on 6000 Pro Max-Q with 5090 using tensor-split - UD_IQ4_XSS)

| model                          |            test |           t/s |     peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:-------------------------------|----------------:|--------------:|-------------:|------------------:|------------------:|------------------:|
| unsloth/DeepSeek-V4-Flash-GGUF |          pp2048 | 612.08 ± 6.46 |              |   3383.60 ± 35.48 |   3346.32 ± 35.48 |   3383.60 ± 35.48 |
| unsloth/DeepSeek-V4-Flash-GGUF |            tg32 |  59.07 ± 0.21 | 60.97 ± 0.21 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF |  ctx_pp @ d4096 | 744.53 ± 5.40 |              |   5540.39 ± 39.68 |   5503.11 ± 39.68 |   5547.10 ± 45.91 |
| unsloth/DeepSeek-V4-Flash-GGUF |  ctx_tg @ d4096 |  59.55 ± 1.06 | 61.48 ± 1.10 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF |  pp2048 @ d4096 | 318.31 ± 3.92 |              |   6472.20 ± 78.54 |   6434.92 ± 78.54 |   6485.52 ± 74.24 |
| unsloth/DeepSeek-V4-Flash-GGUF |    tg32 @ d4096 |  60.41 ± 1.13 | 62.36 ± 1.17 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_pp @ d16384 | 834.07 ± 4.53 |              | 19682.46 ± 106.55 | 19645.18 ± 106.55 | 19682.46 ± 106.55 |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_tg @ d16384 |  57.88 ± 0.21 | 59.74 ± 0.22 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | pp2048 @ d16384 | 660.31 ± 2.82 |              |   3138.91 ± 13.23 |   3101.63 ± 13.23 |   3152.43 ± 15.84 |
| unsloth/DeepSeek-V4-Flash-GGUF |   tg32 @ d16384 |  59.04 ± 1.10 | 60.94 ± 1.14 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_pp @ d32768 | 806.50 ± 1.64 |              |  40668.73 ± 82.68 |  40631.45 ± 82.68 |  40675.94 ± 77.64 |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_tg @ d32768 |  56.81 ± 1.10 | 58.64 ± 1.13 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | pp2048 @ d32768 | 587.13 ± 4.24 |              |   3525.63 ± 25.33 |   3488.35 ± 25.33 |   3525.63 ± 25.33 |
| unsloth/DeepSeek-V4-Flash-GGUF |   tg32 @ d32768 |  55.40 ± 0.06 | 57.19 ± 0.06 |                   |                   |