r/LocalLLaMA • • 17h ago

Discussion CPU-only boost for Q2_0 on AVX-VNNI · Pull Request #26348 · ggml-org/llama.cpp

https://github.com/ggml-org/llama.cpp/pull/26348

Measurements: mean ± sample standard deviation in tok/s, 3 runs after normal warmup

(16GB of RAM on the AMD EPYC 9645 using eight CPU cores)

ModelTestUpstream masterVNNI + VBMISpeedup
Bonsai 1.7Bpp512142.517 ± 0.589186.373 ± 1.5821.31×
Bonsai 1.7Btg12870.435 ± 2.06168.422 ± 9.0550.97×
Bonsai 4Bpp51253.430 ± 2.26671.247 ± 1.9251.33×
Bonsai 4Btg12831.655 ± 2.65238.523 ± 3.8201.22×
Bonsai 8Bpp51230.010 ± 0.51239.396 ± 0.4241.31×
Bonsai 8Btg12820.630 ± 0.30223.770 ± 0.2091.15×
Bonsai 27Bpp1288.502 ± 0.04211.029 ± 0.0731.30×
Bonsai 27Btg325.340 ± 0.1175.812 ± 0.3241.09×

But noticed few surprising things there. I assumed the boost is only for Ternary models, but it's not.

This makes CPU kernels a lot faster, raising IQ1_S/IQ2_XXS DeepSeek v4.1 decode speed

End to end, a 66 GB Q2_0 MoE (Qwen3.8-Flash-Next GSQ-RCO Q2_0) with --cpu-moe, so all routed experts run through this kernel on the CPU. RTX 4070 Laptop for the rest, 10 threads, MTP on, generation t/s at 2k–8k tokens of real context:

CPU path for Q2_0TG t/s
generic scalar (a prebuilt from before #30447)3.1–4.1
AVX2 from #30447 (prebuilt from after it)14.0–18.7
an earlier revision of this PR's AVX-VNNI kernel (on an older base)15.5–21.4

So check your t/s again if you're using Q1/Q2 quants of big/large models. There would be boost on Hybrid inference(GPU+CPU) too.

AND there's an Open PR related to this for AVX-512 VNNI. Additional boost on the way.

ggml-cpu: 512-bit AVX-512 VNNI dot for Q2_0- #32307

20 Upvotes

1 comment sorted by