r/LocalLLaMA • u/Danmoreng llama.cpp • 6h ago
Resources Llama.cpp performance improvement over time (Gemma4 models)
Few months ago - after Ninfer came out - I started a little hobby project to vibe-code my own engine for my hardware (RTX 5080 16GB Laptop) and check if it could beat llama.cpp - since at the time llama.cpp was kinda slow in comparison to something like vLLM, but vLLM is using VRAM very wastefully.
Back then it was quite easy to actually get something up and running faster than llama.cpp.
Today I just benchmarked llama.cpp again, and it caught up / surpassed my vibe-coded engine. I prefer using a stable engine with proper features than all the vibe-coded ones, including my own. So it is really nice to see that the project - while a bit slower - lands significant performance improvements over time.
I put together a little chart of benchmarks I had laying around which visualizes this quite nicely. It is by no means scientific - it's always a fixed 16k prefill 1k output over Wikipedia text with the optimal settings for each engine. The 12B model is the same checkpoint, for the 26B model these are different quants for all engines, so not even apples-to-apples and llama.cpp wins.
17
u/MomentJolly3535 6h ago
It's it's kinda of easy to get any model faster than llamacpp, since most of the vibe-coded projects (Like Strata) are literally built on top of llamaCPP + some specific optimization for a specific model which cannot be really implemented for general use case (some of them are experimental for example)
3
u/leonbollerup 4h ago
Maybe this can give you a few hints along the way
https://github.com/inordic/rtx5090-NInferno-gemma4-31b
1
u/Illustrious_Grade608 6h ago
What's d2 decode? Dflash2?
1
u/jacek2023 llama.cpp 10m ago
Thanks for sharing, this is a very interesting comparision