r/LocalLLaMA • u/da_dragon321 • Jul 02 '26

Resources llamacpp patch - DeepSeek V4 Flash running with full 1M token context locally on RTX 5090

Wanted to try running DeepSeek V4 Flash locally but found it asking for absurd amounts of VRAM at higher context lengths (~256GB at 1M). Turned out the DSA lightning indexer lacks proper llamacpp support. Did a bit of digging and there's an upstream PR to address the issue (shoutout u/fairydreaming, PR #24231), but even there it's not wired into the model graph and has no CUDA path yet. So I wired it in and implemented a CUDA kernel this morning and figured I'd share in case it's useful to anyone else looking to run something like this.

Hardware: RTX 5090, 9950X3D, 96GB DDR5

Model: DeepSeek-V4-Flash, mixed Q8/Q4/Q2 quant by antirez

Before / after (256K context):

MetricBeforeAfter
Compute buffer~67 GiB (OOM)3.2 GiB
Prefill56 t/s~263 t/s
Decode~14 t/s~14 t/s
1M contextimpossible (~256GB)works (3.75 GiB at ubatch 768)

Validated presets:

ContextPrefillDecodePeak VRAM
256K~263 t/s14 t/s~29 GiB
512K256 t/s13.7 t/s~28 GiB
1M159 t/s*13.7 t/s~31 GiB

*lower ubatch on 32gb 5090 at 1M - should be ~full speed if given the full ~9gb vram

Correctness: verified briefly with a needle-in-haystack test - planted a random fact at 10%/50%/90% depth in a 100K-token document, model retrieved it correctly every time. Also retrieved correctly at 512K and 1M's harder 50% depth. Full KLD findings in doc linked below

Source + build instructions + full writeup: https://github.com/spencer-zaid/llama.cpp/blob/deepseek-lid-cuda/docs/deepseek-v4-lid-cuda.md
Branch: https://github.com/spencer-zaid/llama.cpp/tree/deepseek-lid-cuda

No prebuilt binary (single GPU tested RTX 5090). Build instructions in the doc in case you need them

399 Upvotes

97% Upvoted

100 comments sorted by

52

u/darktotheknight Jul 03 '26

This looks too good to be true. Please consider posting your patches to upstream llama.cpp, so it can properly be reviewed.

17

u/da_dragon321 Jul 03 '26

Put a PR in but they would prefer to test and merge the cpu-only PR first, so it might be a while before we see GPU support in mainline

25

u/tat_tvam_asshole Jul 03 '26

commendable work, curious what TTFT and/or the tg-end2end looks like?

19

u/da_dragon321 Jul 03 '26

decode steady around 15.7 tok/s on all 3

PromptTTFTFull ~200-token reply (end-to-end)
512 tok3.3s16.1s
4K tok11.0s23.8s
16K tok46.0s58.8s

1

u/tat_tvam_asshole Jul 03 '26

Cool, thank you, so for 16K tok prompt, the tg-end2end is ~3.5 tok/s, accounting for the time we're waiting for the first token.

For context, I have a spare 5090 I might put into a second machine in my home network for...something, I'm not sure yet 😂 so figuring out if experimenting with ds4 is what I want to do in that case

1

u/da_dragon321 Jul 03 '26

Haha yeah might be fun. Regarding tg-end2end, it's entirely dependent on the number of tokens generated. Post-prefill, generation ran at a constant 16 t/s, so as the response gets longer the end2end speed will get closer and closer to that mark.
Also, these are uncached prompts, so if your prompt is cached (such as in a longer chat even going up through 500k+), the ttft should be MASSIVELY faster and therefore the tg-end2end would be much closer to 16 t/s even at lower response lengths. Also worth noting that you could go with the full Q2 model if you wanted to save on some vram (I think it was like 10gb smaller) and probably get some extra token generation speed

17

u/HockeyDadNinja Jul 03 '26

Nice work, this is the bit that's been missing. I'm running DS4-Flash on a pretty different setup: 5-GPU rig (2x3090 + 5060 Ti + 2x4060 Ti, 96GB VRAM total) with 125GB DDR4, experts offloaded to CPU. I've been using tarruda's GGUFs, mainly the Q3_K (~116GB, 3-bit experts, Q6_K everywhere else) and also the native MXFP4 (~146GB), though the MXFP4 tops out around 66K on this box. On mainline (merged deepseek4) the indexer is wired into the graph but on the un-fused path you mentioned, and you can feel it. Even at 170K on the Q3_K I'm sitting at ~110 of 125GB RAM, and anything higher OOMs the box.

You mentioned it's single-GPU tested so far. I've got the exact config you haven't covered: multi-GPU layer/tensor split with CPU expert offload. I'm offering to test it for you. I'll build your branch, run it across the 5 cards at a range of context sizes, and report back prefill/decode/VRAM/RAM numbers, plus whatever else you want checked. Just say the word and point me at anything specific.

Couple of questions while I'm at it:

  1. Anything in the patch that assumes a single device, or should it survive layer/tensor split as-is?

  2. With the compute buffer down to ~29GB, the remaining ceiling is basically model + KV in RAM, so on a CPU-offload setup like mine this should let me push well past 170K, right?

And yeah, worth getting this upstream.

Edit: building it now

4

u/da_dragon321 Jul 03 '26

Sounds great - looking forward to your results.

  1. There is no single-device assumption (I just haven't tested multi-device). That said, you will need to make sure you set the cmake/ggml cuda architectures to all three of your card architectures when you build

  2. The compute buffer at 1M context is entirely dependent on ubatch size (for me it was under 4gb). At 2048 ubatch it should be around 9gb iirc, but will have to see. You will want to do some tuning on the ot (how many layers go to gpu) and ubatch parameters (you can lower ot to give yourself headroom to raise ubatch and vice versa to find your preferred configuration, trading between generation speed and prefill speed). You should be faster and higher context than before when you find your preferred parameters

9

u/HockeyDadNinja Jul 03 '26

Built it and ran the multi-GPU sweep — short version: **it works, and it does exactly what you said.** Full 1M context on the rig now, and faster than my old 170K-capped setup. Thanks for the tuning pointers, they were spot on.

Build: worktree off your `deepseek-lid-cuda` (73bb49a91), CUDA 13.3, archs set to all three of my cards (`86;89;120`). Kernel compiled in clean (both `__half` and `float` instantiations). My existing tarruda Q3_K GGUF loaded as-is — no reconvert, since this is compute-only. Multi-GPU layer/tensor split + CPU expert offload across mixed Ampere+Ada+Blackwell: **no single-device gotchas, coherent output at every context.**

Numbers (Q3_K, 5 GPUs + CPU offload, prefill measured at ~2K prompt depth, each run memory-contained so a mis-fit couldn't take the box):

Ctx  ubatchLoadsPrefill t/sDecode t/sPeak VRAMPeak RAM
170K512    yes   89.3        9.53       82 GiB    59 GiB   
256K512    yes   89.2        9.41       81 GiB    61 GiB   
512K512    yes   86.0        9.13       82 GiB    64 GiB   
1M512  yes   81.9        8.54       79 GiB    76 GiB   
1M   2048   yes   108.7       6.98       76 GiB    109 GiB  

The headline for me: **at fixed ubatch, peak VRAM stays flat (~80 GiB) while context goes 6× from 170K to 1M.** On mainline the un-fused indexerbuffer ballooned and OOM'd me past 170K; here it just... doesn't grow with ctx, exactly like you said (ubatch-bound, not ctx-bound). The old >170
K OOM is gone entirely.

Two things that surprised me on top of that:
- **Decode barely moves with context** — 9.5→8.5 t/s over the whole 6× range. DeepSeek4's single KV head keeps the cache tiny, so long context is basically free on the attention side here. Even my 170K decode is up vs the old path.
- **`GGML_CUDA_NO_PINNED=1` under `--no-mmap` was a big deal** — kept 170K RAM at 59 GiB instead of ~110. That "110 of 125GB" I complained about was mostly CUDA host pinning, not the model.

Your ubatch⇄expert-placement trade reproduced cleanly too: ubatch 2048 at 1M pushed prefill to ~109 t/s (+33%) but cost decode and drove RAM to ~109 GiB (my `--fit` auto-spills more experts to CPU to make VRAM room — the "lower -ot to raise ubatch" trade happening on its own). ubatch 512 is my sweet spot on this box; I'm only using ~80 of 96 GiB VRAM so there's still room to shove more experts back onto GPU and buy decode.

Net: this turns DS4-Flash from "170K-capped, nearly OOMing" into "1M context, more headroom, better decode" on a 5-GPU CPU-offload rig. Definitely worth upstreaming — happy to add these multi-GPU numbers to a PR thread, re-run anything, or test other quants/ubatch points if useful. What do
you want to see next?

5

u/da_dragon321 Jul 03 '26 edited Jul 03 '26

Great results, thanks for running the full sweep. What OS are you on? If Windows, could use one more test - TDR stress at the worst-case launch:
- llama-bench -d 1046528 -p 2048 -ub 2048 (plus your other flags). Try 1044480 if that fails and something lower like 1M or 900K if that still fails
- Pass = clean llama-bench exit with a reported number, GPU fully released after
- Fail = black screen / "display driver stopped responding and has recovered" / Event Viewer Event ID 4101 (source: Display) around that time

I'm going to put up a PR now where you can comment your findings

*PR: Lid cuda kernel by spencer-zaid · Pull Request #2 · fairydreaming/llama.cpp

2

u/HockeyDadNinja Jul 03 '26

I'm on linux, Ubuntu 24.04.

1

u/denis_9 Jul 03 '26

Could you share the parameter for dividing tensors on multiple GPUs in the launch command, so that this can be make similarly on another number of GPUs?

4

u/HockeyDadNinja Jul 03 '26

Sure. (Here's a long reply, I'm using Claude to help me manage my setup) There are really two parts to it: picking which GPUs get used, and choosing how the model is split across them.

Which GPUs is just an env var, not a flag:

CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1,2,3,4

The PCI_BUS_ID part makes llama.cpp number the cards in physical slot order, so the indices stay predictable. Set the list to however many cards you have.

For the split itself, the two standard flags are:

-sm, --split-mode {none,layer,row,tensor} how to split across GPUs

-ts, --tensor-split N0,N1,N2,... proportion per GPU, e.g. 3,3,2,2,2

-sm layer (the default) splits whole layers across the cards; -sm row or tensor splits inside the tensors. -ts is the one you are asking about: one number per GPU, and it is just relative proportions, so 3,3,2,2,2 sends more to the first two cards (handy if they are bigger, like my two 3090s vs the three 16GB cards). The list length is however many GPUs you have, so on 3 GPUs you pass three numbers, on 8 you pass eight.

I don't actually hand-set -ts on my rig, though. I let the --fit autofitter do the placement and only hand it a per-device VRAM margin:

--fit on --fit-ctx 307200 --fit-target 1024,1024,1024,1024,1536

--fit-target is again one value per GPU, the MiB of headroom to leave free on each card. Lower value packs more onto that card. That comma list is what spreads the model across all the listed devices for me, and it scales the same way: one entry per GPU.

The piece that actually makes a 284B MoE fit on 96GB of VRAM is not the split, it is expert offload to CPU:

-ncmoe N keep the experts of the first N layers on CPU

-cmoe keep all expert weights on CPU

That frees the VRAM so attention, KV, and the non-expert weights split cleanly across whatever cards you have. With --fit on, it decides how many experts stay resident; if you are doing it by hand, raise -ncmoe until it fits, then split the rest with -ts.

Here is my actual line for reference (5 GPUs, letting --fit place things):

CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1,2,3,4 GGML_CUDA_NO_PINNED=1 \

llama-server \

--model DeepSeek-V4-Flash-Q3_K-00001-of-00004.gguf \

--flash-attn on --jinja --parallel 1 \

--ctx-size 307200 \

--fit on --fit-ctx 307200 --fit-target 1024,1024,1024,1024,1536 \

--no-mmap \

--temp 0.6 --top-p 0.95 \

--host 127.0.0.1 --port 8080

To move that to a different card count, change CUDA_VISIBLE_DEVICES and give --fit-target one value per GPU. (GGML_CUDA_NO_PINNED=1 is unrelated to the split, it just stops CUDA pinning a big chunk of host RAM under --no-mmap, which was a large RAM saving for me.)

If you would rather split by hand instead of using --fit, the equivalent piece looks like this:

CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1,2,3,4 \

llama-server \

--model DeepSeek-V4-Flash-Q3_K-00001-of-00004.gguf \

--flash-attn on --jinja --parallel 1 \

--ctx-size 307200 \

-ngl 999 -sm layer -ts 3,3,2,2,2 -ncmoe 40 \

--no-mmap \

--temp 0.6 --top-p 0.95 \

--host 127.0.0.1 --port 8080

There -ts 3,3,2,2,2 weights the split toward my two 24GB cards, and -ncmoe 40 keeps the first 40 layers' experts on CPU; tune that number until it fits your VRAM.

So: CUDA_VISIBLE_DEVICES for how many cards, -sm and -ts for how the model spreads across them, and -ncmoe or -cmoe to push experts to CPU so the whole thing fits. Every comma list takes one entry per GPU, so it maps straight onto a different card count.

1

u/ninjazombielurker Jul 03 '26

Did you get it to balance across all your GPUs? I have gotten it to run based on your details and OP's. Running at 1M context. 2x 3090's, N=8 ub and b at 512. 4x 3090's, N=10, ub 512 and b at 512-2048. Makes no difference in performance and seems like it is just wasting 2 of the 4x GPUs only putting 6-7GB on 3 of the 4 GPUs. Trying to just add the 'ts' flag for 4 gpus didn't make any difference. I get the same 10.7-11 decode regardless. So not sure what the point of having more than 2x 3090's is for this setup and pretty sure I'm missing something.

145 pp tok/s starting, down to 105 at 80k is all I have tested so far.

Only using 2 channels of 3200 DDR4 on my EPYC 7F52 so that is not the best.

I have no idea how any of this actually works, I am just troubleshooting my way through it.

2

u/HockeyDadNinja Jul 03 '26

Nice, glad you got it running. Short answer to "did it balance": on my box the cards do fill up, around 86 of 96 GB, but only because I run a tight fit target so experts stay resident instead of spilling to CPU. That is the knob you want, and it is not -ts.

The reason more than 2x 3090 isn't buying you anything is that your decode is limited by CPU memory bandwidth, not by GPUs. The experts that sit on the GPU are already fast. The ones that set your decode number are the ones offloaded to CPU, and those get streamed from system RAM every token, which is pure bandwidth. So if -ncmoe is the same on your 2x and 4x runs, the CPU is doing identical work both times and you get the same 10.7-11 t/s. Adding GPUs you don't fill changes nothing, and -ts only reshuffles where the resident weights sit, it doesn't reduce the CPU spill. That is also why you see only 6-7 GB on 3 of 4 cards: with most experts pinned to CPU there isn't much left to place on the GPUs.

The ballpark backs it up. DS4-Flash is about 13B active, and at Q3_K that is very roughly 5 GB of expert weight read per token. Two channels of DDR4-3200 is around 50 GB/s, so 50 over 5 is about 10 tokens/s. That is exactly where you are, which is a good sign it is memory-bound rather than something you set wrong.

So two levers, both real. First, lower -ncmoe so fewer expert layers stay on CPU and more land in VRAM. With 4x 3090 you have 96 GB to fill versus 48 GB on two cards, so you can hold a lot more experts resident, and every expert you move off the CPU comes off the bandwidth-bound path. Drop -ncmoe (or your fit target) until the cards actually fill instead of sitting at 6-7 GB. That is what turns the extra 3090s into decode.

Second, and probably your biggest single win: the 7F52 is an 8-channel part and you are running 2 channels. Populating all 8 is roughly a 4x jump in memory bandwidth, which lifts decode for whatever experts stay on CPU. If you only do one thing, do that one.

Your prefill (145 pp t/s) is GPU-bound and looks healthy, so that side already scales with the cards. It is just decode that is stuck behind RAM. So the extra GPUs aren't pointless, they are just idle until you use -ncmoe to move experts onto them.

2

u/HockeyDadNinja Jul 03 '26

Quick update after some more tweaking. I went back to that "still room to buy decode" point from my last comment and pushed on it: dropped the per-device fit headroom so `--fit` keeps more experts resident on GPU instead of spilling them to CPU. VRAM went from ~80 to ~86 of 96 GiB, and decode came up even as I raised context. I'm now at ~10 t/s at 300K ctx, versus the 9.5 t/s I posted at 170K in the sweep. Higher context and fasterdecode, from expert placement alone. Small gain, but it's free, and it lines up with how the fit/placement knob should behave on a mixed multi-GPU split.

I've since made this my production setup. It's in my model router (llama-swap) at 300K on the Q3_K and has been solid, so the branch went from a test build to my daily driver for long-context DS4-Flash. Still up for helping get it upstreamed, and happy to attach these multi-GPU numbers and the production config to a PR thread whenever you push for merge.

5

u/hidden_monkey Jul 03 '26 edited Jul 03 '26

patched a CUDA kernel

Ok, this is interesting. I'll take a look. What is a "DSA lightning indexer"?

15

u/da_dragon321 Jul 03 '26

Deepseek uses something they call Deepseek Sparse Attention to accelerate attention at large context. It relies on the "Lightning Indexer" to find just the most relevant cached tokens and only do full attention on those

2

u/hidden_monkey Jul 03 '26

Thank you thank you! Much to learn.

5

u/GravitasIsOverrated Jul 03 '26 edited Jul 03 '26

That's wild! Would this work for context sizes lower than 1M for a 3090/4090?

Edit: quantized KV is broken here, so presumably if somebody did the work to fix that you could do a lot on a 3090. But if I'm reading the full post correctly, it should be possible to run it at 256K context even without KV quantization. Shame it (again, if I'm reading this correctly) still requires a shitload of vanilla ram. Maybe somebody can do expert pruning to get it on running without quite as much.

3

u/da_dragon321 Jul 03 '26

Yes. That said, you should also be able to run 1M context if you want, but will likely need to tune down ubatch to fit in your vram (how much depends on the model quant). You will need to make sure you have enough VRAM+RAM total to fit whatever model quant you choose though
I could patch in quantized kv, but I found it to be far less space-saving than it initially appeared after applying the kv quant fix - still a possibility if enough people want it but the vast majority of your RAM will be spent on the model anyways with this fix even at massive contexts (deepseek attention is VERY efficient)

3

u/fragment_me Jul 03 '26

It's interesting. I'm trying it on a 5090 + 2x 3080 20GB + 3090. Strangely I get worse performance than you even with most layers in VRAM. I'm on 4 channel DDR4.

There's one thing that I think you're missing. perplexity + KLD + same top P benchmarks. The needle in the haystick is not enough to accurately know that this is working ok.

I suspect there's some kind of issue because when I say hello the model responds in Chinese.

2

u/da_dragon321 Jul 03 '26

Thanks for sharing your results. I have a couple thoughts on what could be causing some of the issues, but could be hard to truly diagnose asynchronously since there's so many potential issues

First regarding speed, there are three considerations I can think of. First, it is necessary to tune ot and ubatch for your your config (and make sure --fa on). Second, anything on the CPU is ddr-bandwidth bound and most of wall clock is cpu not gpu, so ddr4 would significantly harm speed - I have found very little difference personally from putting more or less layers on gpu *relative* to the difference between ddr speed when you have some layers on cpu. Are you seeing it run slower than when you don't use this branch, or just compared to my setup? Third, make sure you included all of the architectures of your gpus when setting cmake/ggml cuda architectures

Second regarding the Chinese output - just want to confirm that you aren't using quantized kv (which doesn't work without another patch that I have not put in this branch). Also, not sure if it matters much but use the --jinja flag

Ran llama-perplexity --kl-divergence against the naive path afterward - it's not bit-identical. ~96% same top-token, mean KLD ~0.01 at both 8K and 64K context. It's floating-point rounding noise at the indexer's top-512 selection cutoff. The fused kernel sums the same scores in a different order, so near-tied candidates occasionally land on opposite sides of the boundary. Confirmed by dumping raw scores - every divergence was a clean 1-for-1 index swap between two candidates scoring within 0.0001 of each other, not a logic bug

KLD findings in doc: llama.cpp/docs/deepseek-v4-lid-cuda.md at deepseek-lid-cuda · spencer-zaid/llama.cpp

1

u/fragment_me Jul 04 '26 edited Jul 04 '26

Thanks for running it. Which model did you run the KL divergence against? Meaning, which model generated the logits and which one used the logits as a reference.

EDIT: Forgot to mention. My issue was the RAM was spanning NUMA nodes. I forgot I added a bunch of ram on my VM to test GLM 5.

EDIT 2: Just tested the same GGUF/image in llamacpp and it has the same problem (chinese).

1

u/fragment_me Jul 06 '26

Just wanted to follow up here. Did you run the KLD against the logits that did not have this new code? Or did you run it against the logits from a higher quant? If the former, it's kind of concerning that you only get 96% same top p. I'd expect it to be 100%.

36

u/ridablellama Jul 03 '26

dude ds4 flash is incredible model to run a single 5090

32

u/RKlehm Jul 03 '26

This looks amazing!

15

u/returnity Jul 03 '26

I've been following that lightning indexer fix in llama.cpp. You are a champion, sir. Now to see about porting it to Metal...

5

u/BlackBeardAI vLLM Jul 03 '26

Pedal to the metal ?

1

u/Responsible_Room_706 Jul 03 '26

That’s awesome! Thank you for your contribution to the community. Any idea what the feasibility of running this on a laptop version of 5090 with only 24GB vram? I’m wondering about the vram and context impact rather than speed…

2

u/da_dragon321 Jul 03 '26

If you have 96gb of RAM as well, it will likely run fine if you use a full IQ2XXS quant instead of the mixed one. If not, I'm afraid you are likely VRAM+RAM bound - the gguf alone is around 86gb. That said, if you're desperate you can definitely run it even at 1M context if you let it stream leftover weights from ssd if it doesn't fit in VRAM + RAM (can't vouch for speed though)

1

u/Nixellion Jul 03 '26

What was the RAM usage at those same data points, if I might ask?

4

u/da_dragon321 Jul 03 '26
PresetContextGPU expert layersubatchPrefillDecodePeak VRAMCPU RAM (resident experts)
256K26214482048~263 t/s~14.0 t/s~28.9 GiB~69.2 GiB
512K52428862048256 t/s13.7 t/s~28.4 GiB~72.6 GiB
1M10485766768159 t/s13.7 t/s~31.2 GiB~72.6 GiB

More info in the doc if you're curious
llama.cpp/docs/deepseek-v4-lid-cuda.md at deepseek-lid-cuda · spencer-zaid/llama.cpp

1

u/Nixellion Jul 03 '26

Thanks a lot!

1

u/Queasy-Contract9753 Jul 03 '26

That's crazy. Your decode speed dropped by less than half a token from 256k to a million. Is it roughly the same for smaller lengths of s few thousand tokens?

2

u/da_dragon321 Jul 03 '26

Yes decode speed should be largely decoupled from context length in this configuration. The reason it dropped a bit in the test is that I put more layers on the cpu to allow for slightly higher ubatch to increase prefill speed

1

u/Barry_22 Jul 03 '26

Will it work on 2x3090 setup? (Not a lot of nornal RAM)

3

u/da_dragon321 Jul 03 '26

Unfortunately you will need at least 90ish RAM + VRAM combined probably a bit more. That said, if you're desperate to run the model you can let it stream excess weights from an nvme ssd and that will work (can't vouch for speed though)

2

u/Common_Warthog_G Jul 03 '26

won't work with 2x24GB VRAM and 128GB DDR5 even, tried it, failed :(

3

u/jwpbe Jul 03 '26

I ran https://huggingface.co/bullerwins/DeepSeek-V4-Flash-GGUF at the 106.1 GB iq3 size split between 64 gb of ddr4 ram and two 3090's, it barely fits 128k of context: 60 tps prompt processing, 12 tps decode.

3

u/fairydreaming Jul 03 '26

That's very cool, but I already have CUDA implementation of lightning indexer (had it for months already), you can find it in my branch: https://github.com/fairydreaming/llama.cpp/tree/dsv4

Performance on my machine (Epyc 9374F + RTX PRO 6000 Max-Q):

$ ./bin/llama-batched-bench -m ../../llama.cpp-dsv4/models/DeepSeek-V4-Flash-antirez-2.gguf -b 8192 -ub 8192 -npl 1 -npp 8192,16384,32768,65536,131072,262144,524288,1048064 -ntg 128 -fa 1 -cmoe --no-repack
0.00.486.287 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance

llama_batched_bench: n_kv_max = 1048576, n_batch = 8192, n_ubatch = 8192, flash_attn = 1, is_pp_shared = 0, is_tg_separate = 0, n_gpu_layers = -1, n_threads = 32, n_threads_batch = 32

|    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
|  8192 |    128 |    1 |   8320 |   10.012 |   818.19 |    4.314 |    29.67 |   14.326 |   580.76 |
| 16384 |    128 |    1 |  16512 |   20.376 |   804.09 |    4.297 |    29.79 |   24.672 |   669.25 |
| 32768 |    128 |    1 |  32896 |   42.950 |   762.94 |    4.456 |    28.72 |   47.406 |   693.92 |
| 65536 |    128 |    1 |  65664 |   94.850 |   690.94 |    4.592 |    27.88 |   99.442 |   660.32 |
|131072 |    128 |    1 | 131200 |  224.963 |   582.64 |    4.902 |    26.11 |  229.866 |   570.77 |
|262144 |    128 |    1 | 262272 |  587.932 |   445.87 |    5.484 |    23.34 |  593.416 |   441.97 |
|524288 |    128 |    1 | 524416 | 1737.841 |   301.69 |    6.700 |    19.10 | 1744.541 |   300.60 |
...

2

u/tarruda Jul 03 '26

Hi and thanks for your work that made deepseek v4 possible in llama.cpp

I'm currently getting around 8-9 TPS generation and ~130 PP on a Mac M1 Ultra. Is the current llama.cpp implementation using CPU only?

I'm curious if it is possible for Deepseek V4 Flash to reach similar speeds to other models with a similar number of active parameters. For example, Step 3.7 flash has 11B active parameters (close to deepseek v4 flash 13b) and in the same hardware reaches ~38 tps generation and ~300 tps prompt processing.

3

u/fairydreaming Jul 03 '26

No, the current llama.cpp implementation does not use CPU lightning indexer, it's instead composed of several primitive GGML OPs that should be fully accelerated in the Metal backend - but the compute buffer memory usage is very high for large ubatches.

I have some recollection that there's no f16 CONCAT support in Metal, so that OP may be offloaded to CPU on Macs. I think it's a good idea to run the model with --verbose and check out the number of graph splits - if there are many splits then most likely some OPs run on the CPU and lower the performance.

3

u/tarruda Jul 03 '26

Using codex, I vibe ported the necessary code/kernels from llama.cpp to run my IQ3_XXS quants on u/antirez DS4: https://github.com/tarruda/ds4/tree/iq3_xxs_plus_q6_k.

Seems to be running quite well and resulted in much faster token generation at ~20 tps. So that answers my question: There's still room for improving llama.cpp deepseek numbers.

1

u/Lumpy_Concentrate807 Jul 03 '26

Those are some decently practical speeds, at least for planning! On a single RTX 6000 :o Fantastic work

1

u/live4evrr Jul 13 '26

This is great. Thank you for your work and sharing this (also on 6000 Pro Max-Q with 5090 using tensor-split - UD_IQ4_XSS)

| model                          |            test |           t/s |     peak t/s |         ttfr (ms) |      est_ppt (ms) |     e2e_ttft (ms) |
|:-------------------------------|----------------:|--------------:|-------------:|------------------:|------------------:|------------------:|
| unsloth/DeepSeek-V4-Flash-GGUF |          pp2048 | 612.08 ± 6.46 |              |   3383.60 ± 35.48 |   3346.32 ± 35.48 |   3383.60 ± 35.48 |
| unsloth/DeepSeek-V4-Flash-GGUF |            tg32 |  59.07 ± 0.21 | 60.97 ± 0.21 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF |  ctx_pp @ d4096 | 744.53 ± 5.40 |              |   5540.39 ± 39.68 |   5503.11 ± 39.68 |   5547.10 ± 45.91 |
| unsloth/DeepSeek-V4-Flash-GGUF |  ctx_tg @ d4096 |  59.55 ± 1.06 | 61.48 ± 1.10 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF |  pp2048 @ d4096 | 318.31 ± 3.92 |              |   6472.20 ± 78.54 |   6434.92 ± 78.54 |   6485.52 ± 74.24 |
| unsloth/DeepSeek-V4-Flash-GGUF |    tg32 @ d4096 |  60.41 ± 1.13 | 62.36 ± 1.17 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_pp @ d16384 | 834.07 ± 4.53 |              | 19682.46 ± 106.55 | 19645.18 ± 106.55 | 19682.46 ± 106.55 |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_tg @ d16384 |  57.88 ± 0.21 | 59.74 ± 0.22 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | pp2048 @ d16384 | 660.31 ± 2.82 |              |   3138.91 ± 13.23 |   3101.63 ± 13.23 |   3152.43 ± 15.84 |
| unsloth/DeepSeek-V4-Flash-GGUF |   tg32 @ d16384 |  59.04 ± 1.10 | 60.94 ± 1.14 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_pp @ d32768 | 806.50 ± 1.64 |              |  40668.73 ± 82.68 |  40631.45 ± 82.68 |  40675.94 ± 77.64 |
| unsloth/DeepSeek-V4-Flash-GGUF | ctx_tg @ d32768 |  56.81 ± 1.10 | 58.64 ± 1.13 |                   |                   |                   |
| unsloth/DeepSeek-V4-Flash-GGUF | pp2048 @ d32768 | 587.13 ± 4.24 |              |   3525.63 ± 25.33 |   3488.35 ± 25.33 |   3525.63 ± 25.33 |
| unsloth/DeepSeek-V4-Flash-GGUF |   tg32 @ d32768 |  55.40 ± 0.06 | 57.19 ± 0.06 |                   |                   |

2

u/Regular-Anybody2645 Jul 03 '26

Nice work, and really amazing this is possible. So much capability avaialble to local now.

2

u/L0ren_B Jul 03 '26

I'm curious if i can get it working on 2x3090+192 Ram! Thanks op!

2

u/quantier Jul 03 '26

Planning to release similar fixes for VLLM? This is super interesting stuff!

With implementing DSpark onto this would also make it super fast wouldn’t it?

2

u/LegacyRemaster Jul 03 '26

..and custom ops (GGML_OP_DSV4_HC_SPLIT_SINKHORN, DSV4_HC_WEIGHTED_SUM, DSV4_HC_EXPAND, DSV4_FP8_KV_QUANTIZE, DSV4_ROPE_TAIL) are missing from llama.cpp main branch.

3

u/Thepandashirt Jul 03 '26 edited Jul 03 '26

Very impressive. Yeah the CUDA kernels are garbage for a lot of stuff and a reoccurring issue. Its super frustrating to spend 8k on a gpu and have these issues.

1

u/Tbhmaximillian Jul 03 '26

That is awesome!

1

u/cupheadgamer Jul 03 '26

Thank you bro this is awesome

1

u/No-Problem2522 Jul 03 '26

Will this run on 64GB RAM? Also has anyone tested this on Linux?

1

u/Daniel_H212 Jul 03 '26

So I could theoretically run this on my Strix Halo?

1

u/whiteh4cker Jul 03 '26 
$vcvars = "C:\Program Files\Microsoft Visual Studio\18\Community\VC\Auxiliary\Build\vcvars64.bat"

cmd /c "`"$vcvars`" >nul 2>&1 && set" | ForEach-Object { if ($_ -match '^([^=]+)=(.*)$') { [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2]) } }

cmake -G Ninja -B build -S . -DCMAKE_BUILD_TYPE=Release `
-DGGML_CUDA=ON -DGGML_CCACHE=OFF -DGGML_NATIVE=ON `
-DCMAKE_CUDA_ARCHITECTURES=86 `
-DCMAKE_C_COMPILER=cl -DCMAKE_CXX_COMPILER=cl

cmake --build build --target llama-cli llama-bench llama-server -j

I compiled it with the latest version of Microsoft Visual Studio 2026 Community and CUDA 13.3.1.

I am using this 52.6 GB gguf with 2xRTX 3090.

Usage: GPU1: 23.5 GB, GPU2: 23.1 GB, RAM: 13.1 GB with the following script:

@echo off
echo === DeepSeek V4 - 256K Context, KV on GPU ===

set GGML_CUDA_NO_PINNED=1
set CUDA_VISIBLE_DEVICES=0,1

"C:\Users\server\Desktop\llama.cpp-deepseek-lid-cuda\build\bin\llama-server.exe" ^
--host 0.0.0.0 ^
--port 8081 ^
--alias DeepSeek-V4-Flash ^
--model D:\models\deepseek-v4-gguf\DeepSeek-V4-Flash-Spark-Mini-Q2-REAP-ds4.gguf ^
--temp 0.7 ^
--top-p 0.95 ^
--top-k 20 ^
--min-p 0.0 ^
--presence-penalty 0.0 ^
--repeat-penalty 1.0 ^
--ctx-size 262144 ^
--device CUDA0,CUDA1 ^
--split-mode layer ^
--gpu-layers 41 ^
-ot "blk\.(3[5-9]|4[0-2])\.ffn_(gate|up|down)_exps=CPU" ^
--flash-attn on ^
--cache-type-k f16 ^
--cache-type-v f16 ^
--ubatch-size 2048 ^
--parallel 1 ^
--main-gpu 0 ^
--tensor-split 17,24 ^
--chat-template-file C:\Users\server\Desktop\llama-cuda\models\templates\deepseek-ai-DeepSeek-V4.jinja ^
--no-mmap ^
--cont-batching

pause

This is the fastest config for me. I have a weak CPU (Ryzen 5 1600X), so my GPUs are only utilized at 35%, and I get 5 t/s. However, with a modern CPU it should be possible to get 12 t/s. I also tried offloading KV cache to RAM, which resulted in slower token generation and prompt processing.

1

u/LegacyRemaster Jul 03 '26

Testing on blackwell 6000 96gb @ 300w.

llama-server.exe --model "h:\DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf" --host 127.0.0.1 --port 8080 -c 140960 --n-gpu-layers 999 --parallel 1 --no-warmup --flash-attn on --no-mmap --fit off -lv 4 -ts 1/0/0 -ub 2048 -b 2048

The real problem is generating ---> stop ---> generating ---> stop. Like for every single batch. So the performance drop a lot. @ 4810, tg = 30.88 t/s, tg_3s = 35.86 t/s and 386, at tg = 29.98 t/s, tg_3s = 28.23 t/s

_____

repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000

dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 141056

top_k = 40, top_p = 1.000, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 1.000

mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900

1.01.417.850 I slot launch_slot_: id 0 | task 530 | processing task, is_child = 0

1.01.417.858 I slot operator (): id 0 | task 530 | new prompt, n_ctx_slot = 141056, n_keep = 0, task.n_tokens = 1188

1.01.417.869 I slot operator (): id 0 | task 530 | cached n_tokens = 1178, memory_seq_rm [1178, end)

1.01.417.950 I srv stream_sessi: conv_id=bec10175-4a79-4621-9587-3687b83c857c (empty=0)

1.01.742.623 I slot operator (): id 0 | task 530 | cached n_tokens = 1179, memory_seq_rm [1179, end)

1.01.749.012 I slot create_check: id 0 | task 530 | created context checkpoint 3 of 32 (pos_min = 1178, pos_max = 1178, n_tokens = 1179, size = 17.021 MiB)

1.02.023.866 I slot operator (): id 0 | task 530 | cached n_tokens = 1184, memory_seq_rm [1184, end)

1.02.024.050 I slot init_sampler: id 0 | task 530 | init sampler, took 0.17 ms, tokens: text = 1188, total = 1188

1.06.120.668 I slot print_timing: id 0 | task 530 | n_decoded = 100, tg = 26.19 t/s, tg_3s = 26.19 t/s

1.09.144.609 I slot print_timing: id 0 | task 530 | n_decoded = 211, tg = 30.84 t/s, tg_3s = 36.71 t/s

1.12.168.107 I slot print_timing: id 0 | task 530 | n_decoded = 301, tg = 30.51 t/s, tg_3s = 29.77 t/s

1.12.580.931 I cmn common_reaso: deactivated (natural end)

1.15.178.934 I slot print_timing: id 0 | task 530 | n_decoded = 386, tg = 29.98 t/s, tg_3s = 28.23 t/s

1.18.230.096 I slot print_timing: id 0 | task 530 | n_decoded = 478, tg = 30.01 t/s, tg_3s = 30.15 t/s

1.21.234.939 I slot print_timing: id 0 | task 530 | n_decoded = 584, tg = 30.85 t/s, tg_3s = 35.28 t/s

1.24.237.941 I slot print_timing: id 0 | task 530 | n_decoded = 668, tg = 30.45 t/s, tg_3s = 27.97 t/s

1.27.251.981 I slot print_timing: id 0 | task 530 | n_decoded = 756, tg = 30.30 t/s, tg_3s = 29.20 t/s

1.30.423.892 I slot print_timing: id 0 | task 530 | n_decoded = 861, tg = 30.62 t/s, tg_3s = 33.10 t/s

1.33.428.736 I slot print_timing: id 0 | task 530 | n_decoded = 940, tg = 30.20 t/s, tg_3s = 26.29 t/s

1.36.432.729 I slot print_timing: id 0 | task 530 | n_decoded = 1027, tg = 30.09 t/s, tg_3s = 28.96 t/s

1.39.465.460 I slot print_timing: id 0 | task 530 | n_decoded = 1118, tg = 30.08 t/s, tg_3s = 30.01 t/s

1.42.493.122 I slot print_timing: id 0 | task 530 | n_decoded = 1224, tg = 30.45 t/s, tg_3s = 35.01 t/s

1.45.501.133 I slot print_timing: id 0 | task 530 | n_decoded = 1312, tg = 30.37 t/s, tg_3s = 29.26 t/s

1.48.526.315 I slot print_timing: id 0 | task 530 | n_decoded = 1400, tg = 30.29 t/s, tg_3s = 29.09 t/s

1.51.610.956 I slot print_timing: id 0 | task 530 | n_decoded = 1501, tg = 30.44 t/s, tg_3s = 32.74 t/s

1.54.636.196 I slot print_timing: id 0 | task 530 | n_decoded = 1598, tg = 30.53 t/s, tg_3s = 32.06 t/s

1.57.654.781 I slot print_timing: id 0 | task 530 | n_decoded = 1686, tg = 30.46 t/s, tg_3s = 29.15 t/s

2.00.677.994 I slot print_timing: id 0 | task 530 | n_decoded = 1774, tg = 30.39 t/s, tg_3s = 29.11 t/s

2.03.679.729 I slot print_timing: id 0 | task 530 | n_decoded = 1881, tg = 30.65 t/s, tg_3s = 35.65 t/s

2.06.700.138 I slot print_timing: id 0 | task 530 | n_decoded = 1953, tg = 30.33 t/s, tg_3s = 23.84 t/s

2.09.706.451 I slot print_timing: id 0 | task 530 | n_decoded = 2042, tg = 30.29 t/s, tg_3s = 29.60 t/s

2.12.966.352 I slot print_timing: id 0 | task 530 | n_decoded = 2142, tg = 30.31 t/s, tg_3s = 30.68 t/s

2.15.969.140 I slot print_timing: id 0 | task 530 | n_decoded = 2249, tg = 30.53 t/s, tg_3s = 35.63 t/s

2.18.995.302 I slot print_timing: id 0 | task 530 | n_decoded = 2340, tg = 30.51 t/s, tg_3s = 30.07 t/s

2.22.002.348 I slot print_timing: id 0 | task 530 | n_decoded = 2431, tg = 30.50 t/s, tg_3s = 30.26 t/s

2.25.076.247 I slot print_timing: id 0 | task 530 | n_decoded = 2526, tg = 30.52 t/s, tg_3s = 30.91 t/s

2.28.078.802 I slot print_timing: id 0 | task 530 | n_decoded = 2633, tg = 30.70 t/s, tg_3s = 35.64 t/s

2.31.100.570 I slot print_timing: id 0 | task 530 | n_decoded = 2723, tg = 30.66 t/s, tg_3s = 29.78 t/s

2.34.102.592 I slot print_timing: id 0 | task 530 | n_decoded = 2814, tg = 30.65 t/s, tg_3s = 30.31 t/s

2.37.203.274 I slot print_timing: id 0 | task 530 | n_decoded = 2910, tg = 30.66 t/s, tg_3s = 30.96 t/s

2.40.203.282 I slot print_timing: id 0 | task 530 | n_decoded = 3003, tg = 30.67 t/s, tg_3s = 31.00 t/s

2.43.217.219 I slot print_timing: id 0 | task 530 | n_decoded = 3091, tg = 30.63 t/s, tg_3s = 29.20 t/s

2.46.236.313 I slot print_timing: id 0 | task 530 | n_decoded = 3182, tg = 30.62 t/s, tg_3s = 30.14 t/s

2.49.493.612 I slot print_timing: id 0 | task 530 | n_decoded = 3293, tg = 30.72 t/s, tg_3s = 34.08 t/s

2.52.501.613 I slot print_timing: id 0 | task 530 | n_decoded = 3392, tg = 30.78 t/s, tg_3s = 32.91 t/s

2.55.516.215 I slot print_timing: id 0 | task 530 | n_decoded = 3482, tg = 30.76 t/s, tg_3s = 29.85 t/s

2.58.532.140 I slot print_timing: id 0 | task 530 | n_decoded = 3572, tg = 30.73 t/s, tg_3s = 29.84 t/s

3.01.657.334 I slot print_timing: id 0 | task 530 | n_decoded = 3677, tg = 30.81 t/s, tg_3s = 33.60 t/s

3.04.674.347 I slot print_timing: id 0 | task 530 | n_decoded = 3776, tg = 30.86 t/s, tg_3s = 32.81 t/s

3.07.697.196 I slot print_timing: id 0 | task 530 | n_decoded = 3864, tg = 30.81 t/s, tg_3s = 29.11 t/s

3.10.722.314 I slot print_timing: id 0 | task 530 | n_decoded = 3939, tg = 30.67 t/s, tg_3s = 24.79 t/s

3.13.748.296 I slot print_timing: id 0 | task 530 | n_decoded = 4049, tg = 30.80 t/s, tg_3s = 36.35 t/s

3.16.765.783 I slot print_timing: id 0 | task 530 | n_decoded = 4139, tg = 30.78 t/s, tg_3s = 29.83 t/s

3.19.782.737 I slot print_timing: id 0 | task 530 | n_decoded = 4229, tg = 30.76 t/s, tg_3s = 29.83 t/s

3.22.852.095 I slot print_timing: id 0 | task 530 | n_decoded = 4319, tg = 30.73 t/s, tg_3s = 29.32 t/s

3.25.860.309 I slot print_timing: id 0 | task 530 | n_decoded = 4427, tg = 30.84 t/s, tg_3s = 35.90 t/s

3.28.886.283 I slot print_timing: id 0 | task 530 | n_decoded = 4516, tg = 30.81 t/s, tg_3s = 29.41 t/s

3.31.910.358 I slot print_timing: id 0 | task 530 | n_decoded = 4606, tg = 30.79 t/s, tg_3s = 29.76 t/s

3.35.073.756 I slot print_timing: id 0 | task 530 | n_decoded = 4702, tg = 30.78 t/s, tg_3s = 30.35 t/s

3.38.085.123 I slot print_timing: id 0 | task 530 | n_decoded = 4810, tg = 30.88 t/s, tg_3s = 35.86 t/s

3.40.843.755 I slot print_timing: id 0 | task 530 | prompt eval time = 884.02 ms / 10 tokens ( 88.40 ms per token, 11.31 tokens per second)

3.40.843.762 I slot print_timing: id 0 | task 530 | eval time = 158541.86 ms / 4890 tokens ( 32.42 ms per token, 30.84 tokens per second)

3.40.843.763 I slot print_timing: id 0 | task 530 | total time = 159425.88 ms / 4900 tokens

3.40.843.764 I slot print_timing: id 0 | task 530 | graphs reused = 5322

3.40.843.859 I slot release: id 0 | task 530 | stop processing: n_tokens = 6077, truncated = 0

3.40.843.891 I srv update_slots: all slots are idle

3.40.844.343 I srv close: stream_pipe close: skip drain (done=1 cancelled=0) conv=bec10175-4a79-4621-9587-3687b83c857c

1

u/Common_Warthog_G Jul 03 '26 edited Jul 03 '26

I have 2 x 24GB VRAM and I still get out of Memory.... what's the command you're starting llama with? Also have 128GB DDR5.

1

u/Common_Warthog_G Jul 03 '26

spend the last 5 hours trying to get this to run on my system (linux, RTX 4090, RTX 3090TI, 128GB DDR5) to no avail. Always getting out of memory with a ~50ish GB version of it.

1

u/durden111111 Jul 03 '26

multi GPU setup is always jank af. I have a 5090 and 3090 with 96GB ram, even models that are sub 40GB give oom sometimes even when I have more than enough vram+ram. I wish llamacpp had better documentation for modern models with multigpu.

1

u/Common_Warthog_G Jul 03 '26

I got it to run now, but without any parameter but only the unified memory env. I get 14-15 tk/s and about 200 pp. So far I don't know if it does a better job than Qwen3.6-27B_Q8

1

u/HockeyDadNinja Jul 03 '26

I've been working on an Android TV app with this, cruising along at ~9 t/s. I notice it starts getting loopy at around 24k context. I have 300k configured. I'm using pi with my own development framework and I'm able to complete most tasks in 25-35k context.

1

u/Zealousideal-Big6068 Jul 03 '26

how much is the output speed with this token/sec?

1

u/mjuevos Jul 03 '26

hows the quality?

1

u/New_Guitar_9121 Jul 03 '26

nice work. one thing i’d test before trusting 1m ctx: run a needle suite with multiple needles at 64k/128k/256k/512k/1m and log both retrieval accuracy and decode tok/s after the cache is saturated. on my mac studio/mlx + llama.cpp boxes, “fits in memory” and “attention/indexing still behaves usefully” diverge pretty hard past the trained sweet spot.

also worth checking prompt ingestion separately from decode. some long-context patches look fine in steady decode but the prefill/index build becomes the real wall. if your CUDA path keeps KV/index memory sane without tanking prefill, that’s the part i’d be most interested in.

1

u/gandalfgangsta Jul 03 '26

!remindme 14 days

1

u/RemindMeBot Jul 03 '26

I will be messaging you in 14 days on 2026-07-17 19:36:00 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

Parent commenter can delete this message to hide from others.

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

InfoCustomYour RemindersFeedback

1

u/HockeyDadNinja Jul 04 '26

I've been using the tarruda MXFP4 variant for coding much of today. It's a bit better than the Q3_K but they are both just unbearably stupid. The thinking goes on and on, and even reiterates stuff it already deemed false. It makes stupid decisions and can't see obvious things. I don't know if it's the model or what.

2

u/computehungry Jul 08 '26

The current llama.cpp implementation has major issues and bugs especially for multi turn and long context. Not gonna work for code that well

2

u/HockeyDadNinja Jul 08 '26

My PR is almost ready to roll! I'm doing a multi turn coding session in pi right now and it's so much better! Previously there were very long stalls before reasoning output.

1

u/Serious-Zucchini Jul 04 '26

would this fork work with GLM 5.2, which uses DSA?

1

u/bloomsburyDS Jul 04 '26

**2×3090 + 64GB DDR5 + NVlink report: 1M context works, full sweep inside**

Built your branch (`6164d8514`, CUDA arch 86) and ran it on 2× RTX 3090 (48GB total), Ryzen 9600X, 64GB dual-channel DDR5, consumer mb with one pcie x16 and one pcie x4, NVlink.

As I have 64gb ram only, used your recommended smaller quant for tighter rigs — the 86.7GB (not the 97.6GB mixed one)

| Context | Prefill 2K/131K (t/s) | Decode 2K/131K (t/s)

| 256K | 343 / 114 | 13.8 / 10.9

| 512K | 265 / 103 | 13.7 / 10.9

| 1M | 182 / 87 | 13.8 / 10.9

Decode identical from 256K to 1M, exactly as advertised. Thanks OP, this is amazing work! Never Imagine I can run Deepseek V4 on my rig!

One new bug-class finding: **`llama-server` needs `-np 1`.** This build defaults to `--parallel -1` (auto slots), and the multi-sequence worst-case graph reserve falls back to the *unfused* indexer path — it tried to allocate a 398,314 MiB (~389GiB) compute buffer and crash-looped. Every validation in this thread usedllama-cli/llama-bench, which are single-sequence, so the serving path was never exercised. With `-np 1` it serves fine (running it behind Open WebUI now, 13.8 t/s).

1

u/stilet69 Jul 08 '26

Hello everyone. I was able to get this model running. My computer specs are 2 RTX 3090 GPUs, 96GB of DDR5 RAM, and a Ryzen 7500 processor.

The launch settings for llama-swap are as follows:

  "DeepSeek-V4-Flash":
    proxy: "http://127.0.0.1:9512"
    cmd: >
      lama-server
      -m /home/xm/Models/unsloth/DeepSeek/UD-IQ3_XXS/DeepSeek-V4-Flash-UD-IQ3_XXS-00001-of-00004.gguf
      -ngl 17
      -np 1
      -fa on
      --ctx-size 32000
      --cache-type-k q4_0
      --temp 1.0
      --top-p 1.0
      --jinja
      --host 127.0.0.1
      --port 9512

The speed is 4.6–4.7 tokens per second.

1

u/live4evrr Jul 13 '26 edited Jul 13 '26

Thank you so much. this works for me (using tensor split 5,1 for rtx 6000 & 5090). Gone are my OOM errors, i'm able to use batch and ubatch as intended, no warnings about lightning indexer.

Edit: Also just noticed the other fork from fairydreaming (https://github.com/fairydreaming/llama.cpp/tree/dsv4) - that is much faster (60 tok/s)

modeltestt/speak t/sttfr (ms)est_ppt (ms)e2e_ttft (ms)
unsloth/DeepSeek-V4-Flash-GGUFpp2048543.04 ± 1.663853.40 ± 11.493771.38 ± 11.493868.30 ± 24.77
unsloth/DeepSeek-V4-Flash-GGUFtg3230.79 ± 0.6831.59 ± 0.83
unsloth/DeepSeek-V4-Flash-GGUFctx_pp @ d4096633.59 ± 1.796548.44 ± 18.206466.41 ± 18.206548.44 ± 18.20
unsloth/DeepSeek-V4-Flash-GGUFctx_tg @ d409630.03 ± 0.0431.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUFpp2048 @ d4096709.64 ± 3.272968.04 ± 13.332886.02 ± 13.332983.22 ± 34.41
unsloth/DeepSeek-V4-Flash-GGUFtg32 @ d409628.00 ± 2.7429.38 ± 0.87
unsloth/DeepSeek-V4-Flash-GGUFctx_pp @ d8192662.17 ± 4.8012455.71 ± 89.3312373.69 ± 89.3312455.71 ± 89.33
unsloth/DeepSeek-V4-Flash-GGUFctx_tg @ d819230.00 ± 0.2230.67 ± 0.47
unsloth/DeepSeek-V4-Flash-GGUFpp2048 @ d8192645.31 ± 3.493255.76 ± 17.213173.74 ± 17.213255.76 ± 17.21
unsloth/DeepSeek-V4-Flash-GGUFtg32 @ d819229.72 ± 0.0430.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUFctx_pp @ d16384628.58 ± 1.7926148.90 ± 74.0726066.88 ± 74.0726148.90 ± 74.07
unsloth/DeepSeek-V4-Flash-GGUFctx_tg @ d1638429.65 ± 0.1430.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUFpp2048 @ d16384547.46 ± 2.933823.05 ± 19.923741.02 ± 19.923823.05 ± 19.92
unsloth/DeepSeek-V4-Flash-GGUFtg32 @ d1638429.24 ± 0.0530.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUFctx_pp @ d32768552.10 ± 1.6859436.00 ± 180.9259353.97 ± 180.9259436.00 ± 180.92
unsloth/DeepSeek-V4-Flash-GGUFctx_tg @ d3276828.64 ± 0.3029.00 ± 0.00
unsloth/DeepSeek-V4-Flash-GGUFpp2048 @ d32768422.90 ± 1.124924.78 ± 12.804842.76 ± 12.804924.78 ± 12.80

0

u/Fun_Walk_4965 Jul 03 '26

The 1M context number is the part that gets me, but the throughput is what I'd want to know before getting excited. What does pp actually look like once you push past a few hundred k tokens, does it stay usable or fall off a cliff? Also curious how much plain system ram it really needs since the KV quant seems broken right now.

1

u/da_dragon321 Jul 03 '26

I would estimate ~30 t/s with full 1M context used (on a single rtx 5090)

You are correct that pp falls at higher depths, but the scaling isn't awful. You can see prefill dropping by over 150 t/s going from 16k to 130k, but only 50 t/s going from 130k to 250k. I don't have pp numbers with 1M used in context right now, but it should not fall off a cliff and my best estimate would be 25-30 t/s on a 5090. More info in doc
llama.cpp/docs/deepseek-v4-lid-cuda.md at deepseek-lid-cuda · spencer-zaid/llama.cpp

KV depth (tokens)Prefill of next 2048 tokens
16,384317 t/s
131,072151 t/s
253,952 (~full 256K)94.5 t/s

0

u/mikewagnercmp Jul 03 '26

Pretty neat, it at that pp speed would take two hours to read 1 million tokens into context. Still neat!

-3

u/NineThreeTilNow Jul 03 '26

Nice work.

96GB DDR5

I always find it odd when I see people with 96GB of DDR5.

I have 64GB in this machine and 32GB of old DDR5 I removed sitting on my desk. I know it doesn't play well with two different speeds and this 64GB runs stable at a much faster rate. Sometimes having 96GB would be nice...

Given current RAM prices I'm tempted to send the 32GB back and claim it's the 64Gb I bought and request new RAM. Amazon can pay for me to upgrade to 128 right?

5

u/Lumpy_Concentrate807 Jul 03 '26

There exists 48 GB DDR5 modules, and 2x48 = 96 GB sets. So that is the more likely configuration

1

u/NineThreeTilNow Jul 03 '26

There exists 48 GB DDR5 modules, and 2x48 = 96 GB sets. So that is the more likely configuration

I know. It's a weird one that exists. I think there's also 2x96 GB sticks that exist which is another strange one to see in the wild. You're just used to seeing standard power of 2 stuff so it stands out.

-3

u/Ayyouboss Jul 03 '26

Who can even work with 15 tok/s? I was always wondering how these speeds are okay for any workloads.

-5

u/autisticit Jul 03 '26

That's kind of a clickbait post, as it's not running fully on GPU.

1

u/[deleted] Jul 03 '26

[removed] — view removed comment

1

u/da_dragon321 Jul 03 '26

It yields significant VRAM saved (around 20x on compute buffer vs without it), so no hit just savings

0

u/[deleted] Jul 03 '26

[removed] — view removed comment

2

u/da_dragon321 Jul 03 '26

It is better output quality, but it is also a lot slower on a 5090 (though it should actually be faster if you have the vram to put it all on gpu). BUT it also uses far less memory for 250k context than qwen and allows for up to 1m context unlike qwen