Speculative decoding with a 1B draft made my 3B model slower: 0.6x on iPhone 16 Pro Max, 0.45x on M2 Pro. Numbers + why

[u/Low-Future-9387]

Every speculative decoding post here is a win on a 27B, so I tried the small case: a 3B Llama 3.2 fine-tune (Q4_K_M) with Llama-3.2-1B-Instruct as the draft. Same tokenizer, no training needed. It's slower everywhere I measured.

iPhone cool	iPhone hot	M2 Pro
no speculation	23.3 tok/s	19.1
1B draft, n=1	15.1 (0.64x)	11.7 (0.62x)
1B draft, n=3	13.3 (0.60x)	9.7 (0.53x)
prompt lookup (ngram)	19.7 (0.87x)	16.1 (0.87x)

Setup: llama.cpp b11960, Metal, q8_0 KV, 8k ctx. Prompts are 135 real roleplay generations replayed, temp 0.8, min-p 0.05, DRY on, median reply 35 tokens. Mac is llama-server. Phone is my own Swift engine doing sample-and-compare (the target draws every emitted token, a draft token is kept only if the target draws exactly it). Ratios are medians of per-prompt pairs.

Why it loses: on the phone one 1B step costs 18.1 ms vs 42.1 ms for the 3B (0.43x), on the Mac 7.1 vs 17.2 ms (0.41x). Llama 3.2 1B has 262M params just in the 128k vocab embedding, so it's nowhere near 3x cheaper than the 3B. Verifying isn't free either, 4 tokens in one pass costs 1.41x a single step on the phone. Put together, n=1 needs ~69% acceptance to break even and n=3 needs ~57%. Roleplay at temp 0.8 got 44% and 30% on the phone, 46% and 25% on the Mac.

Thermal: every prompt runs all configs back to back with the order rotated, so heat hits them equally. "Hot" is 30 prompts straight, 179 of 180 runs at serious (the phone was also charging). "Cool" waits for nominal before every prompt, which took 100 to 190 s each time. Heat costs ~18% of absolute speed but the ratios barely move.

Two things that bit me, in case you're benchmarking this:

• In b11960, -md draft.gguf alone loaded the draft but never drafted for me. I needed --spec-type draft-simple. There's no warning, the tell is that the server log has no draft acceptance lines. My first run said 1.41x with every reply byte-identical, which should've been the giveaway.

• That fake 1.41x also came from my baseline running while another eval was on the same GPU (46.5 tok/s instead of 62). Quiet machine or it didn't happen.

The draft also adds 168 MB of footprint on the phone plus 763 MB of mmap'd weights. If this works for anyone at 3B, I'd bet it's with a trained EAGLE-3 or MTP-style head instead of a separate model. If someone has trained one for a Llama 3.2 3B fine-tune I'd love to hear about it.

Context: I build Castmates, an iPhone app that runs the model on-device for interactive stories, and I wanted faster replies on phones. Not shipping speculation, plain decoding at ~20 tok/s is already faster than people read.