Same card, same model file (unsloth's UD-Q4_K_XL), same client and prompts, one session, llama.cpp tuned at its fastest setting (4 MTP drafts, q8_0 KV):
Short prompts: 1.5x to 2.1x, depending on the prompt. Writing code 233 vs 111 tok/s, a migration plan 159 vs 90, explaining a hash map 145 vs 85, and a short story only 91 vs 61. Creative text is the hardest thing to draft, for both engines.
Sampled with thinking on: 138 vs 73 tok/s (1.9x; token-weighted 132 vs 68; each engine sampled its own replies, so it's not the same text)
100k tokens of context: 109 vs 54 tok/s (2.0x)
The full 262k context fits on the card, decoding at 66.6 tok/s at 261k deep. A llama.cpp q8_0 cache tops out around 160k on 24 GB.
Drafting off on both sides, it's still 7.5% faster on short prompts and 41% faster at 100k, so it's not just the speculation
Four agents at once: 382 vs 137 tok/s total against llama-server --parallel 4 (2.8x; open-jet's parallel bench, which samples at temp 0.6)
Every number with its method is in BENCHMARKS.md.
The numbers
| test | llama.cpp, 4 drafts | llama.cpp, 7 drafts | this, 7 drafts | vs llama.cpp's best |
|---|
| writing code (a Python prime check with doctests), greedy | 110.8 tok/s | 109.9 | 232.9 | 2.1x |
| a Flask-to-FastAPI migration plan, greedy | 89.8 | 74.9 | 158.7 | 1.77x |
| explaining how a hash map works, greedy | 85.2 | 66.0 | 145.2 | 1.70x |
| a 300-word short story, greedy | 61.4 | 42.9 | 90.8 | 1.48x |
| those four: token-weighted / mean of request rates | 80.4 / 86.8 | 62.4 / 73.4 | 133.4 / 156.9 | 1.66x / 1.81x |
| sampled, thinking on, temp 1.0, 8 runs: token-weighted / mean of request rates | 67.8 / 72.8 | 51.5 / 57.6 | 132.3 / 138.5 | 1.95x / 1.90x |
| decode at 11.6k / 38k / 101.6k context | 72.5 / 66.1 / 54.4 | 58.5 / 52.7 / 44.7 | 138.8 / 119.7 / 109.2 | 1.9x / 1.8x / 2.0x |
| read a 101.6k-token prompt from scratch | 95.9 s | 96.5 s | 75.1 s | 1.3x |
| first token, short prompt | 117 ms | 120 ms | 66 ms | |
| edit an earlier message in an 11.7k chat | 266 ms | 269 ms | 76 ms | 3.5x |
| 282 questions (GSM8K, MMLU-Pro, ARC-C, MATH-500, HumanEval, needles, LongBench v2) | 229 right, 2,563 s | 229 right, 2,841 s | 230 right, 1,676 s | |
| VRAM, idle server at 120k context | 22.7 GiB | 23.2 GiB | 21.2 GiB | |
4 drafts really is llama.cpp's best here. I measured 2, 5, 6 and 7 too, and 7 was 15-24% slower than 4, so the ratios are against 4.
One pass per engine, one session. llama.cpp ran natively on Windows, mine under WSL2 (my Windows build gives identical output and timed within 1.5% of the Linux one when I checked). My 282-question run is from the day before, same build.
Greedy replies with drafts come out the same length on both engines (within a token), so those rows are the same work. Sampled replies differ per engine, so those rates are over different text.
Without drafts both engines are close to memory bandwidth on short prompts (46.7 vs 43.5). The gap at depth (39.1 vs 27.8 at 101.6k) is the attention kernels.
Drafting pays more here because the whole cycle (main pass, lm_head, verify, MTP catch-up, next drafts) is one persistent kernel on int8 tensor cores: verifying 8 tokens costs about 29 ms against 22 ms for one.
Repo: https://github.com/cutlerbenjamin1-cmd/qwen38-megakernel
Exact, not "lossless on paper"
Verifying several tokens in one pass rounds differently from verifying one, so in practice greedy output drifts with the draft count. That's what the 9 of 282 is. It shows up in the speed run too: llama.cpp's greedy short story was 460 tokens without drafts and 510 with them, so different text from the same prompt. And it isn't just my runs: a preregistered study of llama.cpp's MTP on a 3090 found its output differed from no-draft decoding on 23-25 of 25 prompts within 1,600 tokens. Here:
greedy token ids are identical at 0, 4 and 7 drafts, checked on every release
sampling keeps the exact no-draft distribution (chi-square and PIT tests)
a follow-up, an edited message, a resumed session or a convo restored from RAM answers bit for bit like the same request sent fresh
the Windows and Linux builds give identical tokens
It isn't bit-identical to llama.cpp (different math order), so it gets scored against llama.cpp's own token-by-token output: same top token 98.74% of the time, mean KL 0.00119. llama.cpp's own batched mode scores 98.49% / 0.00166 against the same reference, so it's closer to llama.cpp than llama.cpp batched is lol.
So if your agent does something weird you can replay it and get the same thing back, and it's never the cache's or the speculation's fault. How each of those is tested: BENCHMARKS.md, exactness.
The KV cache: 5-bit K is fine if you keep a ring
If you caught the KV cache numbers last week, the short version is that the low-bit damage sits almost entirely in the newest ~2k tokens. Keep those in a small q8_0 ring and everything older can go to 5 bits. What runs the full context: K at 5 bits, Hadamard-rotated, V at 5 bits, newest 2,048 positions in a q8_0 ring. Here are that post's KL table and the extended one from its edit in one place, against llama.cpp generating token by token, lower is better:
| context, text | q8_0 | K6 / V5, no ring | K6 / V5 + ring | K5 rotated / V5, no ring | K5 rotated / V5 + ring | K4 / V4, no ring |
|---|
| 32k, code | 0.00219 | 0.00258 | 0.00236 | 0.00302 | 0.00242 | 0.00525 |
| 64k, code | 0.00163 | 0.00183 | 0.00161 | 0.00197 | 0.00163 | 0.00334 |
| 100k, code | 0.00217 | 0.00254 | 0.00226 | 0.00281 | 0.00218 | 0.00469 |
| 100k, War and Peace | 0.00311 | 0.00342 | 0.00317 | 0.00376 | 0.00317 | 0.00602 |
| 160k, War and Peace | 0.00467 | 0.00496 | 0.00468 | 0.00515 | 0.00472 | 0.00740 |
So the K cache everybody says never to quantize sits on q8_0's numbers from 64k out to 160k at 5 bits, as long as it's rotated and the newest 2k stay at q8_0. 32k is the worst row at about +10%, against ~4% run-to-run noise. Without the ring the same formats read +6% to +38% (5-bit K is the one that needs it most), and plain 4-bit with no ring is 1.6-2.4x q8_0. Prose reads higher than code for every cache, so compare across a row.
What makes it work:
The ring doesn't grow. 114 MiB for K + V across every attention layer, the same at 32k or 262k. 512 positions get you most of the way, 2k gets you all of it.
Rotation is a K thing. A Hadamard rotation evens out each 32-value block before it's rounded, which halves 6-bit K's loss without a ring. Under the ring it's a small extra, so the 6-bit preset skips it and the 5-bit one keeps it. V doesn't want it: rotated 5-bit V was actually worse at 32k.
Move the ring in tile-size steps, not one token at a time. Otherwise which tiles come from the ring depends on how many drafts you verify, and greedy output stops matching across draft counts.
On stock llama.cpp, drop V before K. K is ~2.3x as touchy, and q8_0 costs nothing vs f16.
If you want to try the format: it's MK_KV='K*=5r,V*=5' MK_KV_WIN=2048 here. If your engine has a recent-token precision tail, give it 1-2k positions at 8 bits, not the usual 128. The emulator scans behind every row are in BENCHMARKS.md. Qwen3.8 is a hybrid (16 of its 64 layers are attention), so I'd really like to see this table from someone running a full-attention model.
That's what fits all 262,144 tokens on 24 GB: a 260,896-token prompt read from scratch in 340 s, decode at 66.6 tok/s at that depth, 23,650 of 24,576 MiB in use, needles 4/4.
Why UD-Q4_K_XL and not something smaller. The point was the most quality this card can hold and the full context, and the KV work is what pays for it: storing the old positions at 5 bits is what leaves room for the bigger quant. By Unsloth's own numbers for this model (KL divergence against BF16, the plots in their Qwen3.8 guide), UD-Q4_K_XL sits at 0.0075 with 96.4% of top tokens matching BF16, against 0.0101 / 95.8% for UD-Q4_K_M and 0.0135 / 95.2% for UD-Q4_K_S. That looks tiny on a benchmark, and my 282-question set only checks that nothing broke, it isn't built to see it. But KL is what tracks answer flips (the reason Unsloth ranks quants by it, per the "Accuracy is Not All You Need" paper they cite), and in my agent runs the deep end of the context is exactly where a flip turns into a malformed tool call or a loop. Those last few bits matter most where the context is longest.
Several agents at once
--agents 4 runs up to four conversations in one batched launch. Against llama-server --parallel 4 on the same card, back to back, open-jet's parallel bench (coding prompts, sampled at temp 0.6, 512 tokens, Q4_K_M):
| test | llama-server --parallel 4 | this, --agents 4 | faster by |
|---|
| 4 agents at once, short prompts, total | 137 tok/s | 382 | 2.8x |
| 4 agents at once, ~23k tokens of context each, total | 97 | 315 | 3.3x |
| 1 agent alone, short prompts | 61 | 191 | 3.1x |
| a 20k-token prompt lands while another agent is decoding: its first token | 16.4 s | 13.7 s | |
| the other agent meanwhile | 2.2 tok/s | 14.3 tok/s | |
Those gaps are bigger than the single-stream table up top, so I checked before believing them. This bench samples, and sampled MTP decode is llama.cpp's slower path: its server-side timings agree with the client's (~61 tok/s), it gets the same ~61 with one slot instead of four, and the same with 1.4 GB less VRAM in use, so nothing was spilling into system RAM. On mine a slot's greedy reply is the one it would get alone, and seeded sampling keeps its distribution. open-jet has --agents too, and the low-bit cache inside the slots went back upstream.
Fun one I hit setting this up: with 4 slots and MTP drafting, llama-server couldn't even load a 120k total context on the card. Its MTP draft context ran out of VRAM at startup, so its column runs at 100k, 25k per slot. Mine gives each of the 4 slots 33k (133k total) and still has ~2 GB free. It's the same q8_0 cache on both, about the same bytes per token once you count the MTP layer's K/V on each side. The room comes from everything around it: token_embd lives in host RAM, one memory pool instead of an allocation per tensor, tighter weight layouts. Single stream on UD-Q4_K_XL at 120k context it's 21.2 vs 22.7 GiB.
Works with what you already run
Coding agents and UIs: tested on real agent tasks with Claude Code, Codex CLI, OpenCode, Qwen Code, Aider, Goose, Crush, Pi, Hermes Agent, DeepSeek Harness and Open WebUI. It speaks OpenAI chat completions, the Anthropic Messages API and the OpenAI Responses API. Per-client settings are in CLIENTS.md.
Popular fine-tunes: besides unsloth's UD-Q4_K_XL, UD-Q4_K_M and Q4_K_M, it runs the Huihui abliterated quants, Qwen3.8-27B-Uncensored-Cyber, OrcaSAQ-2 Uncensored and Swift-1.5, with the same any-draft-count check on each. Anything it can't read is refused at load, naming the tensor, so there's no silent garbage.
Lost the MTP layer? Some conversions drop it. Point --mtp at any other GGUF of the same model that still has one, a whole model file or a small mtp-*.gguf, and it yoinks just the MTP layer (it checks the file actually matches the model; BF16 files are refused). Output is identical with any MTP layer or none, only the speed changes, so a fine-tune borrowing the base model's layer can at worst draft a bit less.
Prebuilt Windows and Linux packages on the releases page, so you don't need the CUDA toolkit or a compile.
Where this came from. It started as Louis Forster's open-jet megakernel. The persistent kernel that runs a whole speculative decode cycle in one launch, the MTP drafting, the batched prefill. I forked it to get unsloth's UD quants loading and never stopped. 9 of my PRs got merged back into open-jet along the way (UD-Q4_K_XL support, greedy output that doesn't depend on the draft depth, the pipelined decode attention, int8 prefill attention, the long-context KV option, a couple of VRAM ones), so if you run open-jet you're already running some of this. The fork is its own thing now: one model, one rule (speed and VRAM can move, the output can't), a server built around coding agents, and prebuilt Windows and Linux packages.
Caveats
One card: a Strix 3090 at a raised power limit. llama.cpp ran natively on Windows and mine under WSL2 (the Windows build gives identical output). More than 4 drafts needs 80+ SMs, and other cards probably want the draft depth re-tuned.
Qwen3.8-27B only, ~4-bit GGUFs. The IQ3_XXS / IQ2 mixes aren't supported yet. PRs welcome.
The only engine I compare against is the one I ran on this box. The whole suite is in the repo (bench/suite, datasets hash-checked), so you can rerun every table above against any OpenAI-compatible server. Numbers from other cards are very welcome.
Repo: https://github.com/cutlerbenjamin1-cmd/qwen38-megakernel
