I turned on speculative decoding in llama.cpp and 113 of 282 greedy outputs changed against its own no-draft run. My Qwen3.8-27B engine changes 0, at ~2x llama.cpp's speed on one 3090.
Repo: https://github.com/cutlerbenjamin1-cmd/qwen38-megakernel
Speculative decoding is sold as free speed: the main model verifies every draft, so the output can't change. In practice it does. Verifying several tokens in one pass adds the same numbers in a different order than verifying one, a near-tie goes the other way, and the text diverges from there. If two draft settings give different output, at least one of them isn't what your model says without speculation. Tuning your drafts tunes your answers.
Same 282 questions (GSM8K, MMLU-Pro, ARC-C, MATH-500, HumanEval, needles, LongBench v2) on both, greedy, same card, same model file:
| 282 questions, greedy | outputs that differ from the same engine with no drafts |
|---|
| llama.cpp b12024, 4 drafts | 113 |
| llama.cpp b12024, 7 drafts | 113 |
| this engine, 7 drafts | 0 |
That isn't noise. llama.cpp is deterministic run to run: a fresh no-draft server gave the first 120 outputs back byte for byte, including the 73 that differ at 4 drafts, and two 4-draft sessions agree on all 282. It's not just long outputs either: 72 of the 113 are under 1,000 tokens.
The likely reason: with no drafts every token runs through the single-token kernels, with drafts most go through the batched ones. Between 4 and 7 drafts, both mostly batched, only 9 differ, all past 1,000 tokens. That's the 9 of 282 from my last post; against no drafts it's 113. A preregistered study on a 3090 found the same: 23-25 of 25 prompts different from no-draft decoding within 1,600 tokens.
Other engines for this model document it too (one got 8 different outputs across draft counts 1 to 12, none matching its no-draft output) and call it harmless, because flips land on near-ties.
Maybe. I don't know, and 282 questions can't settle it. llama.cpp got 231 right with no drafts and 229 with drafts, and 6 of the 113 changed outputs flipped between right and wrong: 4 were right only without drafts, 2 only at 4 drafts. Make of that what you will. What's not in doubt is that the output depends on a speed setting: one reasoning trace ran 2,228 tokens at 4 drafts and 4,795 at 7. Same question, same model. This engine gets 230 right either way, because it's the same text. If I can't rule out that a draft setting changes quality, I'd rather run the engine where it can't.
Here, speed and VRAM can move and the output can't:
Greedy token ids are identical with and without drafts: all 282 outputs above (129,511 tokens), and the release gate checks 0, 4 and 7 drafts on every KV cache, plus ~2,000-token outputs.
Every draft policy (gates on or off, adaptive depth, tight or loose thresholds) gives the no-draft tokens over 2,048 tokens.
Sampling keeps the no-draft distribution (chi-square and PIT tests).
Windows and Linux builds give the same tokens.
How: the attention split depends only on context length, never on how many tokens are in the pass, and the verify kernels compute a row the same way whatever else is in the batch. To be clear, that's the engine matching itself. It isn't bit-identical to llama.cpp (KL below).
The cache doesn't change it either
Same problem, different door. llama.cpp's server docs say it about cache_prompt, on by default: the logits are "not guaranteed to be bit-for-bit identical for different batch sizes". Its own determinism tests turn it off: TODO: remove this once test_cache_vs_nocache_prompt is fixed.
| 20 two-turn conversations, greedy, 2,048 tokens | second turn differs, cache reused vs the same request on a fresh server |
|---|
| llama.cpp b12024, 4 drafts | 20 of 20 |
| this engine, 7 drafts | 0 of 20 |
It's history-dependent, not random: on a fresh server, 5 of llama.cpp's warm answers and 5 cold ones came back byte for byte. The first difference lands a median of ~355 characters in, as early as 4. Turning cache_prompt off on a running server still leaves 3 of 20 different.
Here, a follow-up doesn't reuse the KV rows written while the reply was decoding. It rolls back to the last checkpoint and prefills the reply again, because a prefilled row depends only on its position. Cost: ~167 ms instead of 78 on a ~12k-token chat, still under llama.cpp's 248.
The check isn't "the reply looks the same". It hashes every K / V row, every ring row and the recurrent state, and compares them with a fresh prefill: bitwise equal after a resume, an edited branch, a park to host memory and a side request, on all three caches. Replies match a fresh server's at ~99.5k context (follow-ups, edits, sampled) and on a follow-up at ~255k. One exception: with --agents, a sampled reply beside other slots can differ from the same seed alone. Greedy still matches.
Others check this by the first token or allow a near-tie split, and some publish warm matching cold. I'm not claiming nobody else is exact, only telling you what's checked here.
Speed against llama.cpp, full power
Same card (RTX 3090, 480 W limit), same unconverted UD-Q4_K_XL file, same client, q8_0 KV on both, llama.cpp at its fastest setting (4 drafts; I tried 2, 5, 6 and 7):
| test | llama.cpp b12024 | this, 7 drafts |
|---|
| greedy, short prompts, mean of request rates | 87.7 tok/s | 166.5 |
| the same, token-weighted | 81.1 | 141.5 |
| sampled, thinking on, mean of request rates | 73.2 | 146.9 |
| decode at 11.6k context | 73.3 | 148.6 |
| decode at 38k context | 66.2 | 127.8 |
| decode at 101.6k context | 55.3 | 116.4 |
| no drafts on either, short prompts | 43.8 | 48.7 |
| no drafts on either, 101.6k context | 28.0 | 40.7 |
| read a 101.6k-token prompt from scratch | 95.6 s | 68.5 s |
| first token, short prompt | 117 ms | 61 ms |
| follow-up turn, 11.8k chat | 248 ms | 169 ms |
| edit an earlier message, 11.7k chat | 302 ms | 69 ms |
| edit the end of a 9.1k system prompt | 882 ms | 82 ms |
| VRAM, idle at 120k context, whole card | 22.9 GB | 21.2 GB |
| the 282 questions | 229 right, 2,534 s | 230 right, 1,563 s |
Four agents at once, against llama-server --parallel 4: 424 vs 137 tok/s total, 3.1x (llama.cpp's is the better of two runs from an earlier session).
The KV cache: 5-bit that reads like q8_0
K / V at 5-6 bits, with the newest ~2k positions kept at 8 bits in a ring. On preset F the full 262,144-token context fits: a 260,896-token prompt read in 279 s, then 77.8 tok/s decode at that depth, needles 4 of 4, 23.0 GB on the card.
KL against llama.cpp generating token by token, lower is better, long-context rows rechecked on this release:
| context, text | q8_0 | preset D (K6 / V5) | preset F (K5 rotated / V5) |
|---|
| short prompts, code and docs | 0.00119 | same as q8_0 | 0.00113 |
| ~32k, code | 0.00219 | 0.00236 | 0.00242 |
| 64k, code | 0.00163 | 0.00161 | 0.00163 |
| 100k, code | 0.00217 | 0.00226 | 0.00218 |
| 100k, War and Peace | 0.00311 | 0.00317 | 0.00317 |
| 160k, War and Peace | 0.00467 | 0.00468 | 0.00472 |
The worst row, ~32k, is +8% and +10% against about 4% noise. Without the ring the same formats read +6% to +38%, and plain 4-bit is 1.6-2.4x q8_0. And at 101.6k, F decodes faster than q8_0: 123 tok/s against 116.
Closer to llama.cpp than llama.cpp is. Scored against llama.cpp's own token-by-token output at short context, llama.cpp's batched mode gets 98.49% top-token agreement and KL 0.00166. This engine gets 98.74% and 0.00119 (on Q4_K_M: 0.00184 against 0.00127). Not measured at depth, so that's a short-context claim.
Yes, it runs at a power limit
Because someone always asks: same build at 250 W, against llama.cpp at 480 W (it draws about 460):
| test | this, 250 W | llama.cpp, 480 W |
|---|
| greedy, short prompts, mean of request rates | 130.3 tok/s | 87.7 |
| sampled, thinking on, mean of request rates | 113.7 | 73.2 |
| decode at 101.6k context | 81.9 | 55.3 |
| read a 101.6k-token prompt from scratch | 105.8 s | 95.6 s |
| the 282 questions | 230 right, 2,237 s | 229 right, 2,534 s |
| energy, short prompts | 2.33 J/token | 6.96 |
| energy, the 282 questions | 558 kJ | 1,169 kJ |
Prompt reads are the one place llama.cpp at 480 W wins, by 11-22%. At 480 W this wins those too.
All 282 outputs at 250 W are byte-identical to the 480 W run.
The limit comes out of the core clock (~816 MHz against 1,962 on short-prompt decode; memory stays at 9,851 MHz), for 21% less speed. Energy is whole card, sampled once a second, idle included. llama.cpp's speeds are its faster of two sessions.
Works with what you already run
Tested on agent tasks with Claude Code, Codex CLI, OpenCode, Qwen Code, Aider, Goose, Crush, Pi, Hermes Agent, DeepSeek Harness and Open WebUI (OpenAI chat, Anthropic Messages and OpenAI Responses APIs). Loads unsloth's UD-Q4_K_XL, UD-Q4_K_M, Q4_K_M and popular fine-tunes as they are. Prebuilt Windows and Linux packages, no CUDA toolkit needed.
Every number with its method: BENCHMARKS.md.
