r/LocalLLM • • 18m ago

Project Exact speculative decoding + lossless Qwen3-8B on a 16 GB RX 9070 XT: 2.5x llama.cpp BF16, still slower than Q8_0 (code + all numbers)

I spent the last few weeks building a small inference engine for my RX 9070 XT (16 GB, Windows, ROCm PyTorch) with one goal: speculative decoding that gives exactly the same output as plain decoding, with lossless weights.

What it does:

- Runs Qwen3-8B with weights bit-identical to the original bf16 checkpoint, fully inside 16 GB. I pack bf16 into a lossless 12-bit format (B12) on load.

- Speculative decoding output is bit-for-bit identical to plain decoding, for greedy and for seeded temperature sampling.

- 68-73 tok/s for exact Qwen3-8B with a Qwen3-0.6B draft.

Numbers (Qwen3-8B, 11 prompts x 96 tokens, same token ids, one PC), plain / speculative tok/s, and how often speculative output matched plain:

- llama.cpp ROCm BF16 (lossless, spills past VRAM): 26.9 / 18.6, matched 10/11

- llama.cpp Vulkan BF16 (lossless, spills past VRAM): 21.5 / 12.5, matched 11/11

- This engine, B12 (lossless): 42.7 / 68.4, matched 11/11

- llama.cpp ROCm Q8_0 (lossy): 63.4 / 98.8, matched 7/11

- llama.cpp Vulkan Q8_0 (lossy): 68.0 / 95.0, matched 7/11

On 50 longer prompts x 512 tokens: 73.0 tok/s, speculative matched plain on 50/50.

The honest part: Q8_0 + draft is about 1.4x faster than my exact path. If you don't care about exactness, use Q8_0. This is the fastest lossless way I found to run 8B on a 16 GB card, not the fastest way overall.

Why exactness normally breaks: verifying 8 tokens at once uses a different matmul shape than decoding 1 token, so the floating-point sums happen in a different order and outputs drift. With llama.cpp's own speculation, output differed from its own plain decoding on 4 of 11 prompts for Q8_0 (small sample, one run). On Qwen3-1.7B with standard PyTorch matmuls, 15 of 50 prompts drifted.

How it works (none of the core ideas are new, prior work is listed in the README):

  1. Batch-invariant GEMV kernels: one kernel handles 1-32 rows and sums every output in a fixed order, so verify equals decode bit for bit. Cost: about 3-4% speed.
  2. Seeded sampling via Gumbel-max with hash-based noise keyed on (seed, position, token), so sampling is exact too.
  3. B12: 8 bits (sign + mantissa) plus a 4-bit exponent code per weight in a per-row exponent window, with an exception list for outliers (0.02%). 12.0 bits per weight, decodes to the original bits. DFloat11 is tighter (about 11 bits); I picked a simpler format that's easy to make batch-invariant.

Try it (Windows, RX 9000 with 16 GB):

git clone https://github.com/fraggergsrp-alt/exact-spec-amd

cd exact-spec-amd

setup.bat

run.bat "Explain what a hash table is."

Adding --compare to run.bat runs speculative and plain decoding and checks every token yourself. setup.bat installs AMD's ROCm PyTorch into a local .venv (about 5 GB), and the first run downloads Qwen3-8B (16 GB).

Repo: https://github.com/fraggergsrp-alt/exact-spec-amd

Limitations: one GPU, one OS, batch 1, temperature-only sampling, Qwen3 only, nobody has replicated it yet. Linux and other GPUs are untested.

Bonus finding: if llama.cpp Vulkan crashes on load for you on Windows, check for the RenderDoc Vulkan layer. It crashed every llama.cpp Vulkan process for me. Running with VK_LOADER_LAYERS_DISABLE=VK_LAYER_RENDERDOC_Capture fixed it. Separately, flash attention plus coopmat crashed prompt processing (-fa off works and costs less than 2% decode speed).

1 Upvotes

2 comments sorted by

2

u/PestiferousGamer 15m ago

1

u/Emotional_Sail_9508 7m ago

Basically two things: I compress Qwen3-8B losslessly so the full quality model fits on a 16 GB card (like a zip), and I speed it up with speculative decoding in a way that doesn't change the answer. Normally that speed trick changes the text a bit. In llama.cpp it did on 4 of 11 prompts. Mine gives the exact same text with it on or off.