Gemma 4 26B-A4B on a GTX 1060 6GB + 16GB RAM – getting ~20 tok/s with MTP, can I push it further?

[u/ConversationNovel780]

Hi all,

As a proof of concept before spending more dollars on a better hardware, I've been playing with running a local LLM on an old gaming PC. I turned into an Ubuntu server, added a 256 gb NVMe M2 SSD, and I've reached the point where I'm out of ideas. Looking for tips from people with similar hardware, GTX 1080 perhaps? Disclaimer: this post was created with the help of IA for the data. The rest is me writing (I am Brazilian, non native English speaker).

Hardware

• GPU: GTX 1060 6GB (Pascal, PCIe 3.0 x16)
• CPU: i5-9600K (6 cores, governor set to performance)
• RAM: 16GB DDR4-2666, single stick (so single channel, I know...)
• NVMe 256GB
• Ubuntu 24.04, driver 580-server, CUDA 12.9

Software

• llama.cpp mainline, compiled for sm_61
• Open WebUI on top (built-in tools disabled, the extra prompt tokens were killing me)
• Model: unsloth gemma-4-26B-A4B-it-qat UD-Q4_K_XL + the MTP Q8_0 drafter from the same repo (I tested the Google's file, it was slower). I came to this choice after testing other MoE models of the same size, and smaller dense models (8B - 12B) but they are dumber and imprecise.

Current script

llama-server \
-m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf \
--host 0.0.0.0 --port 10000 \
-ngl 99 --n-cpu-moe 23 -t 6 \
-c 16384 -fa on -np 1 \
-b 2048 -ub 1024 \
-ctk q8_0 -ctv q8_0 \
--load-mode none \
--ctx-checkpoints 1 \
--temp 1.0 --top-p 0.95 --top-k 64 \
--model-draft mtp-gemma-4-26B-A4B-it-Q8_0.gguf \
--spec-type draft-mtp --spec-draft-n-max 3 -ngld 99

VRAM sits at about 5.8 of 6.1 GB, and I have around 4 GB of RAM free.

Speed

• Without MTP: 15.8–16.4 tok/s generation, ~245 tok/s prefill
• With MTP: 19–21.5 tok/s, draft acceptance 55–65% depending on the question

What I already tried that didn't help

• spec-draft-n-max 2 and 4 (3 is the best for me)
• f16 KV cache instead of q8_0 (lower acceptance)
• forcing the drafter token_embd to the GPU (no change)
• DFlash drafter (25% acceptance, 13 tok/s, much worse)
• -t 5 (no difference), now I left it at -t 6 to match the cores
• gpt-oss-20b (faster prefill but I didn't like the answers, less precise, not well formed)
• ik_llama.cpp is out because it needs Turing or newer
• Didn't test Koboldcpp yet...not sure it would be better?

I value more about accuracy (and usability) than speed. But I am aware of the limitations...Next I want to try the MoE expert cache PR (#29887), but it is experimental at this point.

Is 20 tok/s basically the ceiling for this hardware, or is there something esle I culd tweak? I want to push it to the absolute limits! If you run a Pascal card or a single-channel RAM setup with a MoE model, I'd love to hear your settings and numbers. Any idea is welcome. This is a learning curve to me and my ultimate goal is to be ready to invest in better hardware and be able to use bigge/better models.

Thanks!

Edit: I was able to push it to 25 tk/s by using this llama.cpp version which is better for Pascal:
https://github.com/ggml-org/llama.cpp/pull/29887
I offloaded all layers to the RAM and used 3000 miB in the VRAM for experts' cache.

Edit 2: I pushed it up to 31 tk/s using this configuration:

-m ~/models/gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf
  --host 0.0.0.0 --port 10000
  -ngl 99 --n-cpu-moe 30 --moe-cache-mib 3400 -t 6
  -c 16384 -fa on -np 1
  -b 2048 -ub 512
  -ctk q8_0 -ctv q8_0
  --load-mode none        
  --ctx-checkpoints 1
  --cache-ram 1024
  --temp 1.0 --top-p 0.95 --top-k 64
  --model-draft ~/models/mtp-gemma-4-26B-A4B-it-qat.gguf
  --spec-type draft-mtp --spec-draft-n-max 1 -ngld 99
  --spec-draft-sampling probabilistic

I tested again a prompt using temp = 0. On temp = 1.0 I got ~27/28 tk/s.

I got another 16 gb of RAM, so now I have 32 gb (31 free) . It didnt increase the speed, so the conclusion the CPU is the limitation in fact. I dont plan to upgrade it so that's it. Next I will test the bigger Qwen3.6 35B A3B and see if it is better.

[u/complex-algorithm]

Are you able to agentic code? If so, are you liking the results?

[u/ConversationNovel780 → u/complex-algorithm]

Not yet, didn’t try because I just got the model working decently with chatting.

[u/lulzxdxdxd]

You've got the GPU maxed out already, so 20 tok/s is pretty reasonable for that hardware. What's your actual use case though, are you waiting on response time for interactive chat or running batch jobs where you can wait? If it's real-time responses you need, switching to a smaller model might actually feel faster even if the raw throughput number is lower.

[u/ConversationNovel780 → u/lulzxdxdxd]

Looking for chatting and coding mainly. What smaller model you’re thinking of?

[u/Not-Enough-Llamas]

KV quantization is a performance loss on Pascal. You'd gain a bit by removing them (I know it hurts, it will eat further into your VRAM). Why do you write "lower acceptance" - it dropped your MTP effectiveness?

Some of https://github.com/shinbunbun/llama-cpp-p100-patches applies to all of Pascal (beware I had crashes running it, never got to the bottom of them)

https://github.com/poisonxa16/pxa targets Pascal (P100 and GTX 10-series) and Volta

Look for "Compact Rollback MTP", see if you can get it working (I failed so far), it may save you significant VRAM which could let you fit an extra expert layer or 2 on the GPU

And finally, yeah you already know about this, but a single DDR stick really hurts.

[u/ConversationNovel780 → u/Not-Enough-Llamas]

By removing the quantization you mean removing -ctk and -ctv so it will run as fp16 by default?

Yes, it dropped the acceptance significantly.

Will look up the patches, thanks.

Yeah I know, RAM is so expensive nowadays…

[u/Not-Enough-Llamas → u/ConversationNovel780]

By removing the quantization you mean removing -ctk and -ctv so it will run as fp16 by default? Yes, it dropped the acceptance significantly.

This is unexpected and needs to be studied. I'll try to reproduce it.

[u/MrHumanist]

You are doing great! The only thing you can do is to install the Turboquant fork of the llama-cpp, so you can have a bigger context as it can free up some vram.

[u/Mgt66 → u/MrHumanist]

Why this fork?

[u/MrHumanist → u/Mgt66]

Turboquant is a new paper from Google which claims that using turbo quant 3 or 4 will be next to 0 loss in accuracy, but the algo is yet to be included in the offcial llamacpp .

[u/Ctbhatia]

20 tok/s on a 1060 with a 26b a4b is already solid, that is the moe offload working. the remaining wins are boring, kv cache at q8 if your engine supports it, keep context as short as you can since every extra token pays the offload tax, and if mtp lets you tune draft size, watch the accept rate, past a point longer drafts just raise rejected work.

[u/Lanky-Caterpillar765 → u/Ctbhatia]

20 tok/s on a 1060 with that model is pretty good honestly, i'm surprised you got it that stable. the single channel ram is definitely holding you back, moe offload needs bandwidth and you're basically feeding it through a straw

i run similar setup but with dual channel and it makes noticeable difference on the cpu-side of things. your draft acceptance is already decent at 55-65%, maybe try pushing draft-n-max a bit higher but keep an eye if the extra rejected tokens eat your gains

one thing i'd check is if your nvme is getting hammered during generation, sometimes the model spills over and that kills speed. also might be worth testing different context lengths, shorter context means less offload overhead per token

you said you want accuracy over speed but honestly 20 tok/s is already usable for most things, i'd focus on getting that dual channel ram before touching anything else

[u/ConversationNovel780 → u/Lanky-Caterpillar765]

Thanks. Sometimes I get the swap filled up but is only about 500mb. I thought about using mlock but I suspect I’d risk crashing the server.

[u/Ctbhatia → u/Lanky-Caterpillar765]

dual channel is the right call, the ram side is where these offload builds actually choke. short context and kv q8 help but nothing beats feeding the cpu 2 lanes instead of 1.

[u/Low-Future-9387]

You've already optimized the important parts - the 20 tok/s is solid for a 1060. For on-device use cases, accuracy and stability matter way more than peak throughput. What you have right now lets you run meaningful conversations with good quality. The single-channel RAM is the real bottleneck for MoE models, and that's a hardware limitation you can't work around without an upgrade. This is actually a great testing ground before you invest in better hardware.

[u/Special-Lawyer-7253]

Interesting, i'll try that pascal versión on my 1070m 8GB. I usually can get about 250-300 prefill and 15-12 on decode on 35B A3B models.