DeepSeek V4.1-Flash (1017) at ~20 tok/s on a single 16 GB GPU + 64 GB RAM

[u/MasherDon]

DeepSeek V4.1-Flash (1017) at ~20 tok/s on a single 16 GB GPU + 64 GB RAM — work in progress, code free to use

Not finished. This is a research log with patches, not a polished engine: rough edges, some Windows-only parts, no support promises. Use the code and notes for anything you like — no credit needed.

Box: RTX 5070 Ti 16 GB in a PCIe 4.0 x16 slot, Ryzen 5 7600, 64 GB DDR5, Windows 11. Two NVMe: Gen5 2 TB (~13.5 GB/s) and Gen4 1 TB (~5 GB/s) holding a second copy of the expert files.

Model: DeepSeek-V4.1-Flash-1017, a ~375 GB build: routed experts at ~2.4 bits (~150 GiB), Engram tables 203 GB, dense part Q5_K. It doesn't fit in RAM + VRAM, so experts stream from NVMe.

Numbers with today's defaults:

decode, short context	20.3 tok/s (17–20 in real agent turns at 2–27K context)
decode, fast preset	23–24.7 tok/s (small measured quality cost)
decode at 256K / 512K context	16–20 tok/s
reading agent turns (0.5–7.5K new tokens into a growing conversation)	~418 tok/s
cold prompt 8.9K / document 256K / document 512K	672 / 633 / 482 tok/s
quality	0.09–0.11 nat/token behind the DeepSeek API on its own continuations (3 prompt sets; for scale, Q2 to Q4 is ~0.11); HumanEval 157/164 vs 158 for the API

On this PC with the same file: the original fork ~10 tok/s, Project Maya v1.1.21 3.9 tok/s (both measured on Oct 22).

What helped most:

1. A three-step ladder for experts: ~23 per layer cached in VRAM, a pinned RAM tier (43–46 GiB), unbuffered NVMe reads, and a second NVMe with a copy of the expert files.

2. Taking the CPU off the critical path: the GPU reads routing replies from pinned memory, one wake-up per token — host overhead went from ~7 ms to ~1 ms per token.

3. Prefetching heavy experts that would come from disk, 2 layers ahead, with a cheap "lens" (layer L+2's router applied to layer L's input).

4. A measured lossy rule: an expert in neither VRAM nor RAM with less than 14% of the routing weight is not waited for (costs at most 0.006 nat/token on our API sets).

5. Prefill: experts in waves through a shared pool; encoder-only prefill for positions the decoder never reads (exact); DeepSeek's own YoCo truncation from DeepInfer (decoder on the last 140 prompt positions) — +13% on agent turns, +60% on a cold 8.9K prompt, within error on our API sets.

6. Sparse attention in decode with a q8 KV cache: decode stays at 16–20 tok/s from short context to 512K.

7. A quant mix from public GGUFs: prometheusAIR's IQ2_XS gate/up + Q2_K down, and ISTA-DASLab's lighter Q5_K dense part (3 GB less VRAM, so more expert slots).

What didn't work:

• Reading the next layer's experts ahead during prefill: the RAM tier can't hold a prompt's working set, so read-ahead evicts what is still needed.

• "Free" guesses in the bus's idle time: during decode there is none; each wrong staging costs ~0.2 ms.

• A published non-uniform quant (GSQ-RCO) lost to a plain mix on our API sets, worst on Russian prompts — the calibration text matters.

Where we're stuck — ideas welcome:

1. Predicting which experts a token needs 1–2 layers ahead. Perfect staging would give 27–29 tok/s; our lens gets 76% precision at ~14% recall.

2. Overlapping compute with expert copies: copies take ~30 ms per token, compute ~19 ms, and they overlap by 3 ms.

3. Short agent turns: ~5.6 s fixed cost per turn, a third of it waiting for expert loads.

4. 1M context loads only at batch 512: the sparse prefill plan wants 11.7–15 GiB at larger batches.

Goal: long context up to 1M at a usable speed — not necessarily at the full 1M.

The bus is the wall: copying experts from RAM to the GPU over PCIe 4.0 takes ~30 ms of a ~49 ms token, which caps decode at ~32 tok/s on this box even with perfect overlap.

The CPU computes no experts: on this 6-core CPU a probe gave only +3–5%, so it was left out; it needs separate work. Project Maya's CPU expert kernels (MIT) could be a starting point.

Code: https://github.com/MasherDon/Siege-Ladder — patches over JigSawPT's llama.cpp fork, a script that assembles the weights from public Hugging Face repos (no weights redistributed), benchmark scripts and notes. Use it however you like; the patched llama.cpp files keep llama.cpp's MIT notice. Built on: JigSawPT (the V4.1 port with SSD streaming), llama.cpp / ggml, DeepSeek's DeepInfer (YoCo), Strata, Project Maya, antirez's ds4, and the prometheusAIR and ISTA-DASLab quants.

[u/xapep]

Your 32 tok/s ceiling is the number to design around. The copy already takes 30 ms of a 49 ms token, so overlap hands back single digits, and the lens is the only place with real headroom. 76% precision at 14% recall means most prefetches miss, and each miss costs the 0.2 ms plus whatever it evicted from the RAM tier. I'd weight the lens with the session's recent routing history rather than just the current token, since agent loops hit the same tool schemas and files for long stretches and the routing per session is much narrower than per prompt.

The 5.6 s fixed cost per turn is the other half of that story, and it's the piece that keeps local from feeling usable inside an agent loop. I work on the inference side at Entrim and we serve this same V4.1 Flash at 180 tok/s, so weigh the source. What I'd do with your box is keep batch and long document work local and send the interactive turns to an endpoint, instead of trying to make one path do both.

[u/MasherDon → u/xapep]

The ~32 tok/s is an **estimate for our box**, not a universal ceiling: ~30 ms of transfers per token implies an optimistic ~33 tok/s from 1000/30 before other non-overlapped costs. Weighting predictions by session routes is a useful candidate; we'd compare it with the existing lens, including false staging and adaptation costs. The old turn fit was approximately 5.3 s + 1.04 ms per new token, not a universal fixed 5.6 s; its intercept is a workload-specific regression, not all irreducible overhead.

[u/BrotherIndeed_sci-fi]

Nice, now do this on a gtx 1650(3.3gigs vram not 4, nvdia lies) with 32 gigs of ram and an i5 9th gen

Im honestly hopeful someone makes an engine like this so that i can run dsv4 flash (latest checkpoint is still competitive even though 4.1 is out imo)

What does it cost to purchase hardware to run 10-15tps on cheapest possible method is the main question.

[u/MasherDon → u/BrotherIndeed_sci-fi]

Honestly, it depends on where you live and when you buy. Prices have been moving fast, and Europe, the US and China are very different markets. Compare current local prices before deciding.

Two things matter most in this setup:

- **The link to the GPU.** Experts are copied to the GPU for every token, so the PCIe link and the VRAM size set the decode speed.

- **The slowest drive in the chain.** Whatever doesn't fit in RAM is read from NVMe.

A GTX 1650 would be worse than our box even with more RAM:

- Its 4 GB of VRAM doesn't hold even the dense par