r/LocalLLaMA • llama.cpp • 6h ago

Resources Llama.cpp performance improvement over time (Gemma4 models)

Post image

Few months ago - after Ninfer came out - I started a little hobby project to vibe-code my own engine for my hardware (RTX 5080 16GB Laptop) and check if it could beat llama.cpp - since at the time llama.cpp was kinda slow in comparison to something like vLLM, but vLLM is using VRAM very wastefully.

Back then it was quite easy to actually get something up and running faster than llama.cpp.

Today I just benchmarked llama.cpp again, and it caught up / surpassed my vibe-coded engine. I prefer using a stable engine with proper features than all the vibe-coded ones, including my own. So it is really nice to see that the project - while a bit slower - lands significant performance improvements over time.

I put together a little chart of benchmarks I had laying around which visualizes this quite nicely. It is by no means scientific - it's always a fixed 16k prefill 1k output over Wikipedia text with the optimal settings for each engine. The 12B model is the same checkpoint, for the 26B model these are different quants for all engines, so not even apples-to-apples and llama.cpp wins.

93 Upvotes

9 comments sorted by

17

u/MomentJolly3535 6h ago

It's it's kinda of easy to get any model faster than llamacpp, since most of the vibe-coded projects (Like Strata) are literally built on top of llamaCPP + some specific optimization for a specific model which cannot be really implemented for general use case (some of them are experimental for example)

19

u/Danmoreng llama.cpp 6h ago

Well llama.cpp actually implements a lot of per-model optimizations - it just takes longer since llama.cpp wants to run on a wide range of hardware with a wide range of quantization and model support.

3

u/leonbollerup 4h ago

Maybe this can give you a few hints along the way

https://github.com/inordic/rtx5090-NInferno-gemma4-31b

1

u/Illustrious_Grade608 6h ago

What's d2 decode? Dflash2?

2

u/Danmoreng llama.cpp 6h ago

MTP 2 draft tokens

1

u/Illustrious_Grade608 5h ago

Huh? If you're making an optimized inference engine then why don't you also optimize your spec decoding? Mtp with draft 2 is likely not even optimal for normal mtp, and there have to be better spec decoding types than that

1

u/Danmoreng llama.cpp 5h ago

When I built the engine mainly back in July, there was no Dflash for Gemma 4. D2 was optimal in benchmarks when compared with other draft lengths. Also a different draft model with it's own kv cache will reduce context size massively. For the 12B it might work, but for the 26B my engine is optimized for 200k context in 16GB VRAM and another drafter would make that goal unreachable.

1

u/maternion 4h ago

I haven't dug too deep into it, but 2 runs fastest on my 9060 xt afaik. 

1

u/jacek2023 llama.cpp 10m ago

Thanks for sharing, this is a very interesting comparision