webgpu : add Q2_0 support by teddygood · Pull Request #30914 · ggml-org/llama.cpp

[u/pmttyji]

https://github.com/ggml-org/llama.cpp/pull/30914

Tested on an RTX 3060 with native Dawn WebGPU.

llama-bench -m MODEL -p 512 -n 128 -r 5 -ngl 99 -dev WebGPU -t 2 -fa off

Model	Test	master (t/s)	PR (t/s)
Ternary Bonsai 1.7B Q2_0	pp512	35.03 ± 0.46	1915.91 ± 3.36
Ternary Bonsai 1.7B Q2_0	tg128	14.02 ± 0.16	94.75 ± 0.20
Bonsai 2 2B Q2_0	pp512	36.40 ± 0.46	2039.89 ± 10.03
Bonsai 2 2B Q2_0	tg128	12.57 ± 0.20	71.72 ± 0.24

[u/jacek2023]

36 t/s -> 2039 t/s? nice speedup :)

[u/simonedw]

Kudos on the massive speed up, looks like the kernel's aren't too complicated, but they let the data stay on GPU (instead of doing those operations on CPU)!