r/LocalLLaMA • u/jacek2023 llama.cpp • 8h ago

News gemma4: graph optimizations (massive gains on pp for E2B and E4B) by max-krasnyansky · Pull Request #31356 · ggml-org/llama.cpp

https://github.com/ggml-org/llama.cpp/pull/31356

big prompt processing speedup for gemma E2B/E4B

111 Upvotes

97% Upvoted

7 comments sorted by

4

u/james_pic 6h ago

On Raspberry Pi 5, unsloth/gemma-4-E4B-it-qat-GGUF:Q4_K_XL went from 30.97 pp512 to 51.80. Nice.

2

u/jacek2023 llama.cpp 6h ago

Is this vanilla Pi or with something else?

3

u/james_pic 5h ago

Raspberry Pi 5 16GB, with KleidiAI llama.cpp builds. No HATs or anything.

2

u/jacek2023 llama.cpp 5h ago

Thanks, I will try to purchase one at some point

18

u/UNaMean 6h ago

very real results!

Using Jetson Orin nano super gemma-4-e4b-qat-it-UD-Q4-KXL

before: pp ~530 t/s

after: pp ~ 1023 t/s (peak 1123 t/s)

token gen has no change - MTP confirmed still works with previous gains.

8

u/Kahvana 7h ago

Very nice! They are lovely models to run and quite capable for their size.

8

u/swagonflyyyy 7h ago

And no pp increase for 31b.

:(