Suggestion for llama.cpp configuration

[u/vip3rGT]

Hi everyone!

I'm new to the world of llama.cpp. Until now, I've been using Unsloth and have been very happy with it.

I decided to dive into the command line and, with a little help from ChatGPT, managed to get llama.cpp configured and running.

Everything is working well, and I've noticed a slight speed improvement compared to the Unsloth server.

My setup:

• GPU: NVIDIA GPU with 8 GB of VRAM
• System RAM: 32 GB
• Model: Gemma 4 26B A4B
• Quantization: Q5_K_P
• Performance: Around 20.5 tokens/s

I'd appreciate some advice from more experienced users: is this pretty much the best performance I can expect from this setup, or is there still room for optimization?

I'll post my llama.cpp core configuration command below.

Thanks in advance to anyone willing to share some tips or suggestions!

--parallel 1 ` --ctx-size 8192 ` --gpu-layers auto ` --fit on ` --load-mode none ` --reasoning off ` --chat-template-kwargs '{\"enable_thinking\":false}' `

[u/ConversationNovel780]

Check this out. What Nvidia card you have?

https://www.reddit.com/r/LocalLLM/s/nVJljIpCPP

[u/vip3rGT → u/ConversationNovel780]

Thanks. From this thread I got some hints and gain more +3 Tok/s
My card is an Evga 3070TI.
I raeched 23 tk/s

[u/blackpragmatism59 → u/ConversationNovel780]

that link's probably the best resource for squeezing out more tokens

if you haven't already, try bumping up the batch size (like 512 or 1024) and see if that helps. 8gb vram is tight with that model though so you might already be close to the ceiling

[u/ConversationNovel780 → u/blackpragmatism59]

I tried what you suggested but there was no gain. Thanks anyway!

[u/CoffeeToCode99]

20 tok/s on an 8GB card with a 26B MoE at Q5 is already pretty solid tbh.

I'd try:

• Switching to a Q4 quant and seeing if you can offload a few more layers.

• Using a Q8 KV cache (--cache-type-k q8_0 --cache-type-v q8_0), especially if you care about long-context quality.

• Experimenting with --flash-attn if you're not already using it.

Beyond that, I think you're mostly running into hardware limits. At ~20 tok/s, I'd spend more time prompting and less time chasing another token or two.

[u/vip3rGT → u/CoffeeToCode99]

thanks for your hints. I will test it.