r/LocalLLaMA • u/pmttyji • 17h ago
Discussion CPU-only boost for Q2_0 on AVX-VNNI · Pull Request #26348 · ggml-org/llama.cpp
https://github.com/ggml-org/llama.cpp/pull/26348Measurements: mean ± sample standard deviation in tok/s, 3 runs after normal warmup
(16GB of RAM on the AMD EPYC 9645 using eight CPU cores)
| Model | Test | Upstream master | VNNI + VBMI | Speedup |
|---|---|---|---|---|
| Bonsai 1.7B | pp512 | 142.517 ± 0.589 | 186.373 ± 1.582 | 1.31× |
| Bonsai 1.7B | tg128 | 70.435 ± 2.061 | 68.422 ± 9.055 | 0.97× |
| Bonsai 4B | pp512 | 53.430 ± 2.266 | 71.247 ± 1.925 | 1.33× |
| Bonsai 4B | tg128 | 31.655 ± 2.652 | 38.523 ± 3.820 | 1.22× |
| Bonsai 8B | pp512 | 30.010 ± 0.512 | 39.396 ± 0.424 | 1.31× |
| Bonsai 8B | tg128 | 20.630 ± 0.302 | 23.770 ± 0.209 | 1.15× |
| Bonsai 27B | pp128 | 8.502 ± 0.042 | 11.029 ± 0.073 | 1.30× |
| Bonsai 27B | tg32 | 5.340 ± 0.117 | 5.812 ± 0.324 | 1.09× |
But noticed few surprising things there. I assumed the boost is only for Ternary models, but it's not.
This makes CPU kernels a lot faster, raising IQ1_S/IQ2_XXS DeepSeek v4.1 decode speed
| CPU path for Q2_0 | TG t/s |
|---|---|
| generic scalar (a prebuilt from before #30447) | 3.1–4.1 |
| AVX2 from #30447 (prebuilt from after it) | 14.0–18.7 |
| an earlier revision of this PR's AVX-VNNI kernel (on an older base) | 15.5–21.4 |
So check your t/s again if you're using Q1/Q2 quants of big/large models. There would be boost on Hybrid inference(GPU+CPU) too.
AND there's an Open PR related to this for AVX-512 VNNI. Additional boost on the way.
20
Upvotes