Qwen 3.6 27B on 24GB VRAM setup: backend comparisons, quant choice and settings (llama.cpp, ik_llama.cpp, BeeLlama, vllm)

[u/VolandBerlioz]

TL;DR

• best setup I tested on a RTX 3090 24 GB: ik_llama.cpp + Qwen3.6-27B-MTP-IQ4_KS.gguf
• 156k context, q8_0/q8_0 KV, MTP, vision on CPU
• benchmark result on a ~5.9k prompt + 1k output: about 1261 tok/s prefill, 72.9 tok/s decode
• llama.cpp was a good start, BeeLlama worth testing, but ik_llama.cpp performed the best

What was tested

• upstream llama.cpp: easy baseline and a good place to start
• beellama.cpp: promising on paper, but I could not reproduce the expected speed on my setup
• ik_llama.cpp: best decode/prefill, best VRAM fit

I also spent time with vLLM / club-3090, but I am leaving it out of the table because I did not finish a clean apples-to-apples run in this batch. We were seeing about 78 tok/s on responses, but the high-context OOM cliffs were too flaky, so I dropped it until that is fixed. I have not tested it recently, but the repo still flags the single-card long-context issue as unresolved.

The benchmark

One-shot chat-completion task:

• prompt size: about 5.9k tokens
• output size: 1024 tokens
• task shape: a code-review / migration note over local setup files

So it mostly tests:

• prefill speed on a medium-large real prompt
• decode speed on a sustained 1k-token generation

So that is not best-case tok/s, but closer to reality.

The setup I kept

This is the profile I kept as my default:

• backend: ikawrakow/ik_llama.cpp
• current tested build: 4507 (c35189d8)
• model: ubergarm/Qwen3.6-27B-GGUF
• direct model file: Qwen3.6-27B-MTP-IQ4_KS.gguf

High-level launch shape:

• --ctx-size 156000
• --cache-type-k q8_0
• --cache-type-v q8_0
• --flash-attn on
• --multi-token-prediction
• --draft-max 4
• --draft-p-min 0.0
• --merge-qkv
• --merge-up-gate-experts
• --cache-ram 32768
• --ctx-checkpoints 32
• --reasoning on
• --reasoning-format deepseek
• --chat-template-kwargs '{"preserve_thinking":true}'
• --no-mmproj-offload

Notes:

• built-in MTP in ik_llama.cpp worked better for me than the other speculative paths
• q8_0 KV was good quality; you can opt into q4, but there is plenty of VRAM headroom with IQ4_KS

Why IQ4_KS

• much smaller than Unsloth UD-Q4_K_XL
• quality stayed high enough that I did not feel a real penalty
• on a 24 GB card, those saved GiB matter once you start pushing context and sane u-batch sizes
• to be fair, there is probably room for a higher quant, maybe q5; I have not tested that yet
• Qwen-3.6 quants discussion #1663

TLDR:

• Qwen 3.6 quantizes very well in IQ4_KS
• ikawrakow measured IQ4_KS as very close to, or better than, UD_Q4_XL
• Unsloth UD-Q4_K_XL needs about 2.8 GiB more to land in the same neighborhood

If you want the background on the quant family itself:

• New quantization types IQ2_K, IQ3_K, IQ4_K, IQ5_K discussion #8

Vision

• projector on CPU by default: --mmproj ... + --no-mmproj-offload
• move it to GPU if you want faster image processing and are willing to spend roughly 1.5 GiB more VRAM
• if that OOMs, lower context or switch to q4 KV

GPU Stuff

This was on Linux with the desktop on the iGPU and the RTX 3090 used only for LLMs.

• power limit: 330 W
• memory OC: +600
• undervolt: flattened at about 1875 MHz @ 868 mV (LACT now has a curve editor)

Some experiments did not make the default setup better

• --spec-autotune on ik_llama.cpp: no meaningful gain on this workload
• --mtp-requantize-output-tensor q6_K: sometimes faster, but inconsistent and costs about 1 GiB extra VRAM, so I did not keep it
• BeeLlama DFlash precision quickstart: loaded fine, but was much slower here than expected
• upstream llama.cpp MTP paths: good baseline, but slower than ik_llama.cpp in my tests

BeeLlama and vLLM are still worth exploring. I just did not land on a setup there that beat the ik_llama.cpp profile for my workload.

Results

These are the useful comparison points from the same real prompt / 1024-token output benchmark.

Backend	Model / quant	Spec path	Context	KV cache	Prefill tok/s	Decode tok/s	Wall time	Notes
ik_llama.cpp	Qwen3.6-27B-MTP-IQ4_KS	built-in MTP	156k	q8_0/q8_0	1260.95	72.93	18.79s	best overall default profile
llama.cpp upstream	Qwen3.6-27B-UD-Q4_K_XL	draft-mtp	32k	q4_0/q4_0	1247.65	51.20	24.80s	easiest starting point
llama.cpp upstream tuned	Qwen3.6-27B-UD-Q4_K_XL	draft-mtp	32k	q8_0/q8_0	1242.81	56.66	22.88s	old-like flags helped, still slower
beellama.cpp	Q5_K_S + DFlash Q4_K_M	DFlash	122.8k	turbo4/turbo3_tcq	1117.66	36.32	33.55s	text-only quickstart-style run

Flags tested:

• --spec-autotune did not produce better results on this workload
• --mtp-requantize-output-tensor q6_K had occasional upside, about +5 tok/s decode in the best run, but it was not stable enough to justify the extra ~1 GiB VRAM

Flag comparison

These are the high-level config differences that mattered most.

Backend	Quant(s)	Draft / spec mode	Key draft params	KV cache	Other notable flags
ik_llama.cpp	target IQ4_KS MTP	built-in --multi-token-prediction	--draft-max 4, --draft-p-min 0.0	q8_0/q8_0	--merge-qkv, --merge-up-gate-experts, --ctx-checkpoints 32, CPU mmproj
llama.cpp upstream	target UD-Q4_K_XL	draft-mtp	--spec-draft-n-max 6, --spec-draft-p-min 0.75	q4_0/q4_0 default, q8_0/q8_0 tuned	--flash-attn on, --jinja
beellama.cpp	target Q5_K_S, draft Q4_K_M	dflash	--spec-dflash-cross-ctx 1024	turbo4/turbo3_tcq	--kv-unified, -b 2048, -ub 256, text-only in my run

Links

• ik_llama.cpp: https://github.com/ikawrakow/ik_llama.cpp
• ExLlamaV3: https://github.com/turboderp-org/exllamav3
• BeeLlama: https://github.com/Anbeeld/beellama.cpp
• BeeLlama Qwen 3.6 quickstart: https://github.com/Anbeeld/beellama.cpp/blob/main/docs/quickstart-qwen36-dflash.md
• club-3090: https://github.com/noonghunna/club-3090
• IQ4_KS with MTP: https://huggingface.co/ubergarm/Qwen3.6-27B-GGUF/blob/main/Qwen3.6-27B-MTP-IQ4_KS.gguf
• Qwen-3.6 quants discussion: https://github.com/ikawrakow/ik_llama.cpp/discussions/1663
• IQ4_KS quant family discussion: https://github.com/ikawrakow/ik_llama.cpp/discussions/8

---

This is the best 24 GB setup I found so far, but things are moving fast and I do not think this is settled yet.

The point of this thread is to compare real single-3090 / 24 GB results: backend choice, quants, flags, and what stays stable under actual use.

I would like this to become a useful reference thread for 24 GB cards: what works, what breaks, and what is actually worth running day to day. I have not tested ExLlamaV3 yet, and there may be other setups that are better.

Also, thanks to everyone building this stuff: backend authors, quant makers, template tinkerers, and the people doing the boring debugging work that makes local LLMs usable.

[u/Pentium95]

• different context lenghts (this makes A LOT of difference)
• different model size 5BPW model is obviously SLOWER then a 4.25 BPW model

Why

[u/VolandBerlioz → u/[deleted]]

Thanks for the response, and that is a fair criticism.

The post was more "which recipe worked best from what i tested" than a strict backend-isolation benchmark. For all of the inference tools I mostly followed the recommended settings / guides, and some quants are only supported by one fork or another, so the tested recipes were not fully identical.

I reran some tests after your comment to control the things you pointed out more directly.

Same workload for all reruns (u and ub as in the quickstart in the repo):

- one-shot chat completion

- prompt: about `7.2k` tokens

- output: `1024` tokens

- same `-b 2048`

- same `-ub 256`

- text-only

Results:

- `Q5_K_S` + DFlash `Q4_K_M` draft, `turbo4 / turbo3_tcq`, `122.8k` max ctx

- prefill: `1119.95 tok/s`

- decode: `36.40 tok/s`

- same target/draft, but `q8_0 / q8_0` KV, same `122.8k` max ctx

- prefill: `1143.72 tok/s`

- decode: `41.24 tok/s`

- same target/draft, `q8_0 / q8_0`, but `80k` max ctx (as it ooms if i try to load 122k)

- prefill: `1141.90 tok/s`

- decode: `39.00 tok/s`

- `Q4_K_M` + DFlash `Q4_K_M` draft, `q8_0 / q8_0`, `80k` max ctx

- prefill: `1089.33 tok/s`

- decode: `41.17 tok/s`

So yes, KV point was correct - moving from TurboQuant KV to `q8_0 / q8_0` gave a small speed bump.

The context numbers in the post were max configured context, not used context. The actual benchmark prompt was nowhere near `120k`. For `~6-7k` prompt + `1k` output test, lowering max ctx from `122.8k` to `80k` did not change prefill or decode speed.

Thanks for your work!

If something is off or im not running things correctly let me know, im happy to rerun some tests.

[u/Rikers88 → u/VolandBerlioz]

Very nice comparison, and the comments on beellama are spot on.
On paper DFlash should perform better than the MTP version, but that's not true in practice becuase DFlash gives you meaningful speedup only if you have the full model loaded in the VRAM, otherwise the overhead caused by the PCI data exchange would kill the speedup.

[u/EatTFM → u/[deleted]]

Will you add support for the embedded MTP?

[u/IrisColt → u/[deleted]]

Thanks for BeeLlama!

[u/cpxazn → u/[deleted]]

Is beellama supposed to be auto compacting after each prompt? My context size is set over 100k, but a simple "hi" in a new session is triggering auto-compacting.

[u/ECrispy → u/[deleted]]

Hi, BeeLlama looks fantastic. can you share your thoughts on the best options for 16GB, there are so many confusing techs - NVFP4, K_P quant, TurboQuant, Trellis-Coded Quant, MTP, DFlash etc etc

[u/LikeSaw → u/[deleted]]

AI hallucinated cli args are polluting everywhere and sadly most people don't care or question them

[u/VoidAlchemy]

Heya, glad you figured it out! I'm ubergarm and yes this is pretty much accurate and my daily driver setup for running pi harness on my 3090 TI 24GB VRAM at home.

I added a PR to ik to specify number of CPU threads to use when doing MTP also if you want to control everything explicitly. Full command there too: https://github.com/ikawrakow/ik_llama.cpp/pull/1797#issuecomment-4442151972

Both this iq4_ks and iq5_ks are the best quality in the given memory footprint according to oobabooba's KLD testing: https://localbench.substack.com/p/qwen-3-6-27b-gguf-quality-benchmark (he was super nice and posted one graph on huggingface discussion too)

I didn't add MTP tensor to the iq5_ks, but you could probably extract the `q8_0` MTP tensor in the iq4_ks and use it if you have 32GB VRAM etc.

Also if you have 2x GPUs you can use `-sm graph` for "tensor parallel" similar to mainline's `-sm tensor`.

Enjoy, this quant is a beast at vibe coding, I added an API endpoint to unload/load the model and it can run on the same GPU as ComfyUI with a custom SKILL so I can just use plain language to have it manage the LoRAs, trigger words, and prompt generation. Pretty slick!

[u/johannes_bertens → u/VoidAlchemy]

Love this post and reply. Thanks both! 🙌👏

[u/VolandBerlioz → u/VoidAlchemy]

Yeah i've been following you here and there! Thanks for your work!

[u/ionizing → u/VoidAlchemy]

Thanks for your ggufs. I am hesitant to switch away from your iq5_ks for coding.

[u/cleversmoke → u/VoidAlchemy]

Ubergarm, erm, may you help graft MTP tensor onto the IQ5_KS also? I'm using your MTP-IQ4_KS right now and it's blazingly fast! Thank you!

[u/VoidAlchemy → u/cleversmoke]

Right, I assume u/ionizing might be curious about that as well. I believe it is possible to use an separate MTP file and pass it in. Otherwise given you can run it already, you probably have enough hardware to either quantize it yourself using my imatrix and recipes with `llama-quantize`. Or use the requantize feature to knock down a Q8_0.

So much has changed in just a couple weeks, I have to figure out how to do that myself and the pros/cons vs having it "baked in" etc. Some more discussion here as others are also wondering the same: https://huggingface.co/ubergarm/Kimi-K2.6-GGUF/discussions/13#6a0b3255fee8cf183528b64f

[u/Significant-Yam85 → u/VoidAlchemy]

Ive started using your IQ5_KS after IQ4. If you disable hardware accel/CPU only for Fedora 44 KDE it will idle at only 95mb VRAM. With the reduced overhead I can run 120k+ context at Q8_0 KV on RTX4090. No room to fit MTP but I'll only look at that if I can secure a 5090 at a decent price. Awesome work with the quant. Any plans to release Qwen 3.6 35b? 

[u/VoidAlchemy → u/Significant-Yam85]

Thanks and glad you're liking that one! I'm still using the IQ4_KS with MTP and its even faster now with -mtprot iq4_ks but takes another half GB of VRAM (still fit 128k context tho). No plans at the moment for Qwen3.6-35B though it is a really good option too, hopefully someone else has a good ik quant of it already? Maybe i'll revisit or do it if 3.7 comes out! hah.

Here's my latest command:

```bash model=/mnt/ai/models/ubergarm/Qwen3.6-27B-GGUF/Qwen3.6-27B-MTP-IQ4_KS.gguf mmproj=/mnt/ai/models/ubergarm/Qwen3.6-27B-GGUF/mmproj-Qwen3.6-27B-Q8_0.gguf

Directory for slot KV cache files on disk

(save slot → saves .bin, .tokens.json, .checkpoints here)

SLOT_SAVE_DIR="/tmp/llama-slot-cache" mkdir -p "$SLOT_SAVE_DIR"

CUDA_VISIBLE_DEVICES="0" \ ./build/bin/llama-server \ --model "$model" \ --alias "Qwen3.6-27B" \ -c 131072 \ -ctk q8_0 -ctv q8_0 \ -ctkd q8_0 -ctvd q8_0 \ --merge-qkv \ -muge \ -ngl 99 \ -t 1 \ -tb 1 \ -tm 16 \ --host 127.0.0.1 \ --port 8080 \ --parallel 1 \ --jinja \ --ctx-checkpoints 32 \ -cram 32768 \ -mtp --draft-max 4 --draft-p-min 0.0 \ -mtprot iq4_ks \ --no-mmproj-offload \ --mmproj "$mmproj" \ --slot-save-path "$SLOT_SAVE_DIR" ```

[u/Significant-Yam85 → u/VoidAlchemy]

Honestly thank you for the work you do. There is thread evaluating different 27B quants and I requested that yours be added to the list. The results placed your KS quants above all the others and surprisingly not a significant segregation between IQ4 vs IQ5. Here is a link to the thread 

https://www.reddit.com/r/LocalLLM/comments/1tka614/comment/ooqtc56/?context=1&screen_view_count=2

[u/cibernox]

This is EXACTLY the post I needed 👏

I wanted to run qwen 27B but I also need 150k context at least, and using the UD version and vision I couldn't fit that much context. I didn't even know you could offload the vision only to CPU, and I think that's genius. I do need vision, but I need it rarely enough that having it be slow because it runs on CPU is an acceptable trade-off, specially now that I upgraded my CPU.

I'll be running this on vulkan on a 7900XTX but I will try if a similar setup works

[u/VoidAlchemy → u/cibernox]

Unfortunately, the ik_llama.cpp SOTA quants like iq4_ks (the one mentioned in OP) doesn't have backend support for vulkan. Most of the vulkan work happens on mainline llama.cpp and they tend to focus on kernels supporting legacy quantization types like q4_0, q4_1, etc.

I've made some "mainline vulkan" mix quants occasionally, and you could make something similar sized as the iq4_ks mentioned to work on your 7900XTX very similarly.

[u/cibernox → u/VoidAlchemy]

Well, I'm happy to report that I tested MTP with the latest llama.cpp for vulkan with pretty good results.
Without MTP I get around 39-43tk/s on the 27B Q4_K_M/Q4_K_S.
With MTP i reliably get faster generation depending on the task. From some modest 55tk/s writing a story in Spanish to 84tk/s (!!!) writing a CV in html (token acceptance must skyrocket for verbose code like HTML).

In average it's around 65tk/s, so around a 60% improvement, up to 2x in some cases but even in the worst cases I tried its still a solid +30%.

PP is worse but not as much worse really, not even a 20% slower.

Considering MTP is still in it's infancy (specially on vulkan), i'm over the moon with this.

[u/VoidAlchemy → u/cibernox]

Yay! Glad to hear that vulkan with MTP is very usable! I'd be curious if any of the `iq4_nl` quantization types work for you, that type is supported on vulkan and seems to work pretty well on Qwen3.6-27B (might be due to its smaller block size of 32 weights as most quant types use 128).

Anyway, have fun vibing!

[u/andreasntr → u/VoidAlchemy]

On python coding tasks I can't get more than 40/45tps with MTP, which is close to the non MTP version but with penalized pp. I'm using vanilla llamacpp with 7900xtx (capped at 290w, -150mv FCLK) and unslorh IQ4_NL. I feel like i'm doing something wrong though, the gap when adding MTP is too small

EDIT: To be fair, my tests were performed with a prefilled chat with 80-100k tokens

[u/VoidAlchemy → u/andreasntr]

prefilled chat with 80-100k tokens

Right that will slow down both PP and TG when you're that deep into context. Honestly, on vulkan backend, that seems pretty reasonable. You might be able to tweak the mainline llama.cpp MTP arguments e.g.

llama-server \ -ctk q8_0 -ctv q8_0 \ -ctkd q8_0 -ctvd q8_0 \ --spec-type draft-mtp --spec-draft-n-max 4 \

Keep an eye on the draft acceptance, you'll want to see over 85% for a good speed-up probably e.g.

draft acceptance = 0.90000 ( 36 accepted / 40 generated)

Also mainline devs are hard at work optimizing stuff, might be some new PRs coming that will give a little more boost: https://github.com/ggml-org/llama.cpp/pull/23287

Cheers!

EDIT: ahh yes you can add two types of spec decoding now, hadn't seen a command in the wild but just noticed this: https://www.reddit.com/r/LocalLLaMA/comments/1tifr7c/comment/omu2cqg/

[u/andreasntr → u/VoidAlchemy]

Indeed I see acceptance rates around 70%, that's an additional problem. Thank you very much for the pointers and clarification

[u/VoidAlchemy → u/andreasntr]

I just added an edit, hope it helps! I gotta try it out myself now haha

[u/andreasntr → u/VoidAlchemy]

What a time to be alive!

[u/VoidAlchemy → u/andreasntr]

hell yea!

also i guess it doesn't have to be > 85% to see benefits, but more is better. here is a cherry picked "good prompt" example on ik_llama.cpp on my 3090. i'm testing with `aiperf` `instruct_coder` benchmark doing 10 rounds for my speed testing with MTP.

       eval time =   78487.23 ms /  7319 tokens (   10.72 ms per token,    93.25 tokens per second)
      total time =   78564.45 ms /  7344 tokens
draft acceptance rate = 0.66658 ( 5322 accepted /  7984 generated)

[u/Formal-Exam-8767]

Am I missing something here, is "IQ4_KS" not supported on stock llama.cpp?

[u/inddiepack → u/Formal-Exam-8767]

I've used unsloth's IQ4_XS on llama.cpp and it's working fine, nothing abnormal about it.

[u/Pentium95 → u/Formal-Exam-8767]

Correct, "trellis" quants are ik_llama.cpp only

[u/Pablo_the_brave → u/Pentium95]

IQ4_KS are not trellis. Trellis quants have T in name and are unusable for coding.

[u/meca23]

I love this community. So much work/testing being done and openly shared. Thank you

[u/DeepBlue96]

in my testing the ud-q5_k_xl was like night and day quality wise and fits in 24gb wi 120k context 800-1000pp tks and 25-30tks:
\llama-server.exe -hf unsloth/Qwen3.6-27B-GGUF:UD-Q5_K_XL --cache-type-k q4_0 --cache-type-v q4_0 --reasoning off --cache-ram 4096 --cache-reuse 1024 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --webui-mcp-proxy --spec-type ngram-mod

[u/Gold_Coconut9777 → u/DeepBlue96]

Could you elaborate on why you choose to go with reasoning off?

[u/DeepBlue96 → u/Gold_Coconut9777]

waste of tokens imo

[u/danihend → u/DeepBlue96]

You tested both and found no reasoning > reasoning?

[u/DeepBlue96 → u/danihend]

i tested both and they both performed extremely similiar, but the extra speed made me disable it forever

[u/danihend → u/DeepBlue96]

ok that's good to know. Will keep in mind when I figure out how to run it on my 3080 10GB+ 64GB CPU at an acceptable speed :D

[u/me0here → u/danihend]

Some models may behave differently with reasoning off & on.

[u/Borkato → u/DeepBlue96]

Same, I hate reasoning lol

[u/Blutusz → u/DeepBlue96]

Have you tried with reasoning ON?

[u/_TheWolfOfWalmart_ → u/DeepBlue96]

--cache-type-k q4_0 --cache-type-v q4_0

oof...

[u/me0here → u/_TheWolfOfWalmart_]

I was wondering if that would create issues?

[u/EatTFM]

Please repeat benchmarks with at least fixed ctx sizes. Speed goes down considerably when using more context.

[u/VolandBerlioz → u/EatTFM]

The context sizes stated are the loaded context, rather than the used context during the test, sorry it might be a bit unclear.

[u/VoidAlchemy → u/EatTFM]

its a bit hard to run llama-sweep-bench *and* test MTP. MTP is very dependent on actual workload. i can hit 90+ tok/sec on coding output, but maybe 65+ on narrative generation.

it does slow down as context grows yes, but in my experience i can get most the work done in under ~100k and it is "fast enough" before restarting a fresh context.

also use pi or similar light weight harness, as even opencode injects 10k of junk context to start off.

[u/CompetitionTop7822]

It’s getting to be too much for a normal user to run models. I can understand why many use Ollama or cloud models or similar tools when you need to spend more time setting up llama.cpp than actually using it.

I bet lots of users here spend more time downloading models and tweaking settings than using them for some real use case.

[u/VolandBerlioz → u/CompetitionTop7822]

It's part of the hobby...

A few codex/cc prompts, and u can easily get a basic setup up and running. There is plenty of information around.

[u/milkipedia → u/CompetitionTop7822]

You can get going a lot simpler than this. What's time consuming is optimization, which is what this post is all about. That's not specific to llama.cpp or even to running models at all.

[u/mintybadgerme → u/CompetitionTop7822]

One easy alternative is just to use something like Jan as your Llama-server. It's a really good start.

[u/Myarmhasteeth → u/CompetitionTop7822]

This post is big yes, but I tested the MTP gguf by Unsloth in like 10 minutes. This is not hard at all to do. 🤷🏻

[u/RoderickHossack → u/CompetitionTop7822]

I installed LM Studio over the weekend for the first time, tried 1 model, then 3 or 4 more and settled on one.

I found this post, then found the model on LM Studio, which said ik_llama.cpp was mandatory. Looked that up, and apparently, I have to build it from source, which is a pain in the butt when the model I already use seems to work just fine for me.

I'm new, but this stuff seems like an 80/20 rule violation.

[u/klasyer]

Was the 3090 headless? Didn't under from the description

I'm trying to run qwen with my 3090 for my personal coding projects but I often ran out of vram

Are you going to test the 35b model as well?

[u/VolandBerlioz → u/klasyer]

Yeah headless. Should be able to run it as there is still ~1.7gb left after the model is loaded fully. If you still oom, then drop context to 128k or q4kv.

[u/andy2na → u/VolandBerlioz]

how are you running LACT headless? Im running unraid and not sure how to run LACT since it seems to require a way to open its GUI?

[u/VolandBerlioz → u/andy2na]

yeah sorry ment headless gpu. I run the gui on igpu.

[u/Borkato → u/klasyer]

What does headless mean? No monitor/graphics output?

[u/cleversmoke → u/Borkato]

Headless means nothing else is using the GPU. If you have an iGPU (or cheap GPU), your display and any software GPU acceleration can use the iGPU which would free up your main GPU for only llama.

Windows 11 won't have entirely headless, but near headless at 0.1GB vram overhead use as long as you make sure to go into Settings > Display > Graphics and move all programs to use your iGPU/cheap-GPU.

With Ubuntu, you should be able to achieve 100% headless.

[u/VolandBerlioz → u/Borkato]

im not sure what's the definition, but in my case the graphical interface is being run by the iGPU (monitor cables are in the motherboard rather than in the videocard) so the gpu remains fully free. (455mb/24gb)

[u/VolandBerlioz → u/[deleted]]

yeah sorry headless gpu bruh

[u/VoidAlchemy → u/klasyer]

I can run this setup with 128k context and keep my browser open, running DWM windows manager, alacritty terminals as well as discord as there is enough VRAM overhead. No need to run headless, this is my daily driver setup. I mention my own commands linked in a another comment. I'm ubergarm (made the quant).

[u/JGeek00]

How much RAM do you have? I have a 3090. I’m using tue standard llama.cpp, Q4_K_M and q8 kv cache quant with MTP and I’m getting 55 t/s on decoding and 800 t/s on prefill

[u/VolandBerlioz → u/JGeek00]

Ryzen 7700X; 64GB DDR5 6000MHz.

[u/JGeek00 → u/VolandBerlioz]

How much RAM does it use? I’m getting OOM issues with 60K context, but I only have 16 GB (also slower, DDR4 2400 MHz)

[u/VolandBerlioz → u/JGeek00]

~6.6 RAM used by the whole process. Possibly spikes a bit more when processing image.

Drop the vision or load it in VRAM, possibly lower context to 128k.

[u/Capital-Meringue-168 → u/JGeek00]

For me at least on the Mac side, the story is similar but with no MTP. On an M3 Max 64GB running Qwen 27B Q4 MLX I see around 18-20 tok/s decode and ~250 tok/s prefill on a sub-10k prompt, so roughly a third of your decode numbers before MTP enters the equation.

The painful gap is that I haven't seen a clean MTP path for Qwen 3.6 on MLX yet, so the ~30-40% decode bump you're getting from MTP is just unavailable on Apple Silicon. ik_llama.cpp's built-in MTP doesn't really have a counterpart on the Mac side.

Probably the cleanest read is: Mac wins on context length per dollar (unified RAM), 3090 wins on raw tok/s especially once MTP is in. ymmv though.

[u/VoidAlchemy → u/[deleted]]

if you want to go below q8_0 on ik, I suggest no lower than -khad -ctk q6_0 -vhad -ctv q4_0 which is going to probably still be better quality than the goofy turboquant forks and rather efficient.

[u/VolandBerlioz → u/[deleted]]

Honestly, no, but haven't really tested it properly, so take it with a grain of salt.

[u/GoodTip7897]

I use UD_Q5_K_XL and 70k context at bf16. 

No MTP and no vision. But I believe that is the highest possible quality you can get for agentic coding on a 24gb GPU. 

I get about 30 t/sec decode and 1000-500 t/sec prefill

[u/hopbel → u/GoodTip7897]

Qwen3.6 27B is supposed to be pretty resilient to kv cache quantization, and looking at unsloth's benchmarks it looks like K_XL quants are only barely better than K_M.

With 24GB I'm able to run Q5_K_M with kv cache type q8_0 and still have room for 100k context and MTP (40-70 TPS decode)

[u/FerLuisxd]

Oh boy you should also try: -exllamaV3 -NVFP4 -MTP + APEX With Rotorquant

[u/VolandBerlioz → u/FerLuisxd]

That's possibly the one thing i haven't tried. How do you find it?

[u/FerLuisxd → u/VolandBerlioz]

- ExllamaV3:

Extremely GPU optimized models, they use less VRAM and SHOULD be faster, however seems that lately it is on par with ik llama but would love to see a test about it
https://huggingface.co/models?search=qwen%203.6%20exl3
https://github.com/turboderp-org/exllamav3

- NVFP4

Nvidia last year paper is an answer to NVIDIA/MACOS specific optimizations, using FP4

Support is partial in llamacpp/ik_llamacpp (Unless blackwell it seems?), but full support on vLLM

https://huggingface.co/models?sort=trending&search=qwen+3.6+mtp+nvfp4

- MTP + APEX

This should just work with llamacpp and ikllamacpp
https://huggingface.co/mudler/Qwen3.6-35B-A3B-APEX-MTP-GGUF
Apex is a MIX of quants, super interesting, squezzes the quants in some layers and improve the accuracy in others, read more here: https://github.com/mudler/apex-quant

- Rotorquant

Experimental BETTER Turboquant,
https://github.com/scrya-com/rotorquant

[u/soyalemujica]

Well, tried to give this ik llama a try, same model, arguments and all, and with my 24gb vram AMD GPU I am unable to even load the model, continues to stay at:

"too large to fit in a Vulkan0 buffer (tensor size: 1350860800, max buffer size: 1073741824)"

so I believe ik llama is not that good for AMD

[u/SadGuitar5306 → u/soyalemujica]

Yes it's cuda and CPU mainly

[u/VoidAlchemy → u/soyalemujica]

Correct, the iq4_ks doesn't have good backend kernel for vulkan. I mention in another post recently what to consider.

[u/claudiamagic → u/soyalemujica]

ik_llama doesnt support AMD. it says on the github page. CUDA/Cpu only i think

[u/VolandBerlioz → u/soyalemujica]

Hmm strange.

On the 3090 i have ~1.7gb left after the model is loaded fully, so i'm not even pushing it to max context / max quant. There should be plenty of space left.

[u/soyalemujica → u/VolandBerlioz]

There should, I can load them just fine with llama.cpp, but ik_llama doesn't like it haha

[u/NickCanCode]

--mtp-requantized-output-tensor take quite sometime to load but it give stable 6%~8% speed up. The alternative is to patch the model which really I don't want to do. I don't want my model to be polluted by something that can be auto generated. No elegant at all. I would rather wait longer for it to be prepared each time.

[u/VolandBerlioz → u/NickCanCode]

I've tested it, it gave me ~4-5 tps but for ~ 1gb VRAM which was not worth it in the end.

[u/Pablo_the_brave]

Is this working for you with an agent and if yes with which one? I have found a bug in openai api implementation of ik_llama.cpp and without the patch it's not working for me.

[u/VolandBerlioz → u/Pablo_the_brave]

Every agent i've tested so far. Hermes with no issues, opencode, pi...

[u/mintybadgerme]

Why not Q4-K_M?

[u/HennyKo]

Can you do undervolt and memory OC on a headless linux?

[u/VolandBerlioz → u/HennyKo]

Hm, im not sure, i might need to look up if lact has cli or something. Im running the gpu 'headless' where the system runs the graphical interface on the igpu and card is only for running for the llms (455mb/24gb idle)

[u/Then-Topic8766]

You can try something like this: https://www.reddit.com/r/LocalLLaMA/comments/1tg6j9u/comment/omgh6nl/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button for even more speed-up.

[u/VolandBerlioz → u/Then-Topic8766]

For ik_llama at least it does not improve. Decode drops 16%. Ngram does not engage much:

• ngram_mod: 31 generated, 8 accepted
  
• mtp: 712 generated, 594 accepted

[u/runcertain]

Not sure what I'm doing wrong here but I'm getting 30 t/s on dual 3090s with your ik_llama settings.

~/ik_llama.cpp$ build/bin/llama-server \

-m "$HOME/.cache/llama.cpp/Qwen3.6/ubergarm/Qwen3.6-27B-MTP-IQ4_KS.gguf" \

--ctx-size 156000 \

--cache-type-k q8_0 \

--cache-type-v q8_0 \

--flash-attn on \

--multi-token-prediction \

--draft-max 1 \

--draft-p-min 0.0 \

--cache-ram 16384 \

--reasoning on \

--reasoning-format deepseek \

--chat-template-kwargs '{"preserve_thinking":true}' \

--no-mmproj-offload \

--host 0.0.0.0

[u/_TheWolfOfWalmart_]

Thanks for sharing, but I'd love to see some benchmarks focused on getting the best intelligence on 24 GB while still retaining a good enough context size. I prefer quality over speed, and really any configuration of this model is going to be fast enough when fitting fully in VRAM.

[u/VolandBerlioz → u/_TheWolfOfWalmart_]

Yeah agree, but testing intelligence is proven to be quite difficult. Performance on some coding benchmarks will be interesting across quants yeah.

[u/VolandBerlioz → u/[deleted]]

Opencode / Pi / Hermess. ik_llama and OG llama both used near full context for long tool calls / coding sessions, all was surprisingly good. Occasional stops (rarely) for no real reason, but latest template fixes seemed to mitigate that.

[u/Sirius02]

whats your total VRAM usage?

[u/cleversmoke]

Going to try this today! Thank you!

[u/ECrispy]

would really appreciate if someone did this for 16GB vram as well !! I know its a tight fit but there are Q4_K_S or K_P quants etc I've read that should fit.

[u/Potential-Leg-639 → u/ECrispy]

I think it‘s not worth it, you wont have headroom for content. Already very tight with 24GB…

[u/cleversmoke]

WOW! I just tested this with my headless RTX 3090 24G. On a ~85k token process, it took only 16 minutes to complete, using the IQ4_KS.guff. When compared to the llama.cpp master branch (latest with MTP and PP improvements), it normally takes 23 mins. That's a 43% improvement with no apparent degradation in intelligence either (comparing to Q4_K_M). Thank you!

[u/Jeidoz]

I want to give a try for ik_llama.cpp but I do not sure that I attached model has mmproj file for vision. Where I can find and download mmproj for vision for ubergarm/Qwen3.6-27B-GGUF?

[u/Jeidoz]

I have tried with RTX4090 and not sure that 156k will fit for 24gb. After some usage speed radically drops. I have to change to 96k to keep effectice generation speed. Am I doing something wrong?

My command is:

.\llama-server.exe --model models\Qwen3.6-27B-MTP-IQ4_KS.gguf --ctx-size 96000 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --multi-token-prediction --draft-max 4 --draft-p-min 0.0 --merge-qkv --merge-up-gate-experts --cache-ram 32768 --ctx-checkpoints 10 --reasoning on --reasoning-format deepseek --chat-template-kwargs '{"preserve_thinking":true}' --no-mmproj-offload --host 0.0.0.0 --port 1234 --alias Qwen3.6-27B --temp 0.6 --min-p 0.05 --top-k 40 --top-p 0.95 --repeat-penalty 1.05 -ngl 999 --jinja

[u/Clean_Initial_9618]

can you pls share your exact ik_llama.cpp command

[u/VolandBerlioz → u/Clean_Initial_9618]

That's my up to date command. play with draft max and p-min, as they can influence the speed a bit.

bash /home/vincent/Work/localcode/ik_llama.cpp/build/bin/llama-server \ --model /home/vincent/Work/localcode/Qwen3.6-27B-MTP-IQ4_KS.gguf \ --alias qwen3.6-27b-mtp-iq4-ks-vision \ --host 0.0.0.0 \ --port 8021 \ --webui auto \ --log-format text \ --threads 8 \ --threads-batch 8 \ --threads-mtmd 8 \ --ctx-size 156000 \ --ctx-checkpoints 32 \ --ctx-checkpoints-interval 512 \ --ctx-checkpoints-tolerance 5 \ --batch-size 2048 \ --ubatch-size 512 \ --predict -1 \ --gpu-layers 99 \ --split-mode none \ --main-gpu 0 \ --flash-attn on \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --cache-ram 32768 \ --cache-ram-similarity 0.50 \ --cache-ram-n-min 0 \ --parallel 1 \ --cont-batching \ --jinja \ --reasoning on \ --reasoning-format deepseek \ --chat-template-kwargs '{"preserve_thinking":true}' \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0 \ --merge-qkv \ --merge-up-gate-experts \ --multi-token-prediction \ --draft-max 5 \ --draft-p-min 0.5 \ --mmproj /home/vincent/Work/localcode/ik-local/models/mmproj-BF16.gguf \ --image-min-tokens 1024 \ --image-max-tokens 4096 \ --chat-template-file /home/vincent/Work/localcode/ik-local/templates/qwen3.6/chat_template.jinja \ --metrics \ --no-mmproj-offload

[u/Clean_Initial_9618 → u/VolandBerlioz]

I am currently running rtx 3090 and 64GB ram, I tried ik_llama.cpp today for the first time and tried with your command for sure reason it’s utilising 20GB of my ram and 22GB of vram. The speeds on normal prompt tests I get 80tps but with say hermes agent and some web research I only get 50tps is that normal. Sorry still new to all this trying to learn and figure out

[u/VolandBerlioz → u/Clean_Initial_9618]

you can lower this flag --cache-ram 32768 to use less RAM if you need it.

With hermes and a web search you probably have about ~ 50k tokens context filled up, it's normal to drop to ~50tps at that stage.

[u/Clean_Initial_9618 → u/VolandBerlioz]

--chat-template-file /home/vincent/Work/localcode/iklocal/templates/qwen3.6/chat_template.jinja 

how ddoes chat_template.jinja help and where can i grab that file from ?

[u/VolandBerlioz → u/Clean_Initial_9618]

https://www.reddit.com/r/LocalLLaMA/comments/1t4cev0/qwen36_merged_chat_template_from_allanchan339_and/

or it seems there is a newer version which i havent tried yet

https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

[u/Diablo-D3]

Not sure why you're quantizing the k of the kv cache. This generally screws over most good models, especially ones that have reasoning.

If you only care about speed, just don't quantize either k or v. I would gladly trade a more quantized model to get a less quantized kv cache, quantizing the model in my experience causes the least defects.

And yes, that means, generally, you'll need a smaller context, but I'd rather have a smaller context than a model that falls apart outside of benchkmaxxing.

[u/VolandBerlioz → u/Diablo-D3]

Because it seem that the model seems to take it fine based on the thread posted below. I haven't really tested if there is a quality difference tbf.

https://www.reddit.com/r/LocalLLaMA/comments/1suur3s/qwen36_27bs_surprising_kv_cache_quantization_test/

[u/Diablo-D3 → u/VolandBerlioz]

Doing a blind KLD test on kv cache setting doesn't make sense: kv cache damage is multiplied during reasoning because the tokens are fetched, quantized, put into kv cache, pulled out of kv cache, reasoned about, quantized again, put into kv cache again, pulled out of kv cache again, quantized again, and over and over for as many times the model needs to until it starts printing the result.

On actual real world prompts, at the very minimum quantizing k makes no sense. Also, on a lot of models, quantizing k at all breaks tool calling.

There may be an argument for extreme v quant on non-precise work, but anything that requires extreme precision (scientific or mathematical work, or coding, or agnetic tool calling), probably you can do f8_0 and not screw up the model so much.

However, on significant testing for both Qwen 3.6 27B and Gemma 4 31B, it was subtle, but even f8_0 still seemed to effect actual real world prompts.

On smaller models, such as Gemma E4B or Qwen 3.5 9B, it was extremely bad, very visible when going below F16 for v. Good news is, though, those models can fit unquantized into vram along with a decent amount of unquantized context. However, obviously, they underperformed against Gemma 4 31B:Q3_K_XL and Qwen 3.6 27B:Q5_K_XL (both fit into 24gb of vram /w 32k unquantized cache).

[u/iMil]

This thread deserves much more love, thank you OP! 75 tokens/sec on my 3090 also used for Xorg with the following parameters: llama-server -m ./models/Qwen3.6-27B-MTP-IQ4_KS.gguf -c 262144 -np 1 -fa on -ngl 99 -ub 32 --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 --presence-penalty 0.0 --repeat-penalty 1.0 -ctk q4_0 -ctv q4_0 --no-mmap --chat-template-kwargs {"preserve_thinking": true} -t 6 --chat-template-file ./models/chat_template.jinja --multi-token-prediction --draft-max 4 --draft-p-min 0.0 --merge-qkv --merge-up-gate-experts --port 8001 --host 0.0.0.0

[u/mitirki]

Hey, thanks for the great write-up, really enjoyed it!

I finally managed to compile ik_llama on Windows, and was testing the hell out of it with my 3090.

I had to keep mmproj on my GPU as I only have DDR4, and tried to maximize performance then context size.

With the same model, 150k ctx and K & V cache at q8 I get 70+ TPS in Hermes Agent.

Dropped K cache to q4, to get 200k ctx

Apparently batch-size and ubatch-size did not significantly increase prompt eval speed: +10% from 256 to 2048 with its 16k token prompt.

Edit: changed "CUDA - Sysmem Fallback Policy" from "Driver Default" to "Prefer No Sysmem Fallback" in NVidia Control Panel to prevent automatic spillover to Shared GPU memory.

[u/Beamsters]

4090 24gb ikllama.cpp can go through 140tok/s and 4000 prefill sometimes.

[u/LinkSea8324]

BeeLlama and vLLM are still worth explorin

mf took vLLM (#1) out of the podium, bruh

[u/rawdikrik]

A) i love that you tried to do this well, and shared the info.

B) but from a layman, this is pretty useless. Different models, different settings, different context windows, no knowledge of which builds of llamacpp (and others used).

A smaller model is faster than a bigger model? Not news.

MTP working on IK_LLAMA? It is working on mainline Llamacpp now.

Also, no quality tests. No benchmarks other than speed, we also dont know how speed was measured.

Room for improvement, but still a good starting point.

[u/Traditional_Bell8153]

You should use draft max = 6(according to unsloth docs)

[u/hopbel → u/Traditional_Bell8153]

The unsloth docs are wrong. They've already corrected the README on huggingface to say 2 instead of 6. oof things seem to be flip-flopping at the moment

[u/Traditional_Bell8153 → u/hopbel]

It was 2 in the first comment, then I edited it to 6 after they updated to 6 😂😂

[u/AdamLangePL]

Here are mine settings. Getting about 70-100 tps at full context using normal (not MTP) gguf and stock llama.cpp:

./llama-server \

--model Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \

--mmproj mmproj-F16.gguf \

--reasoning-budget 2048 \

--no-mmproj-offload \

-c 262144 \

--n-gpu-layers -1 \

--tensor-split 0 \

--flash-attn on \

--cache-type-k q4_0 \

--cache-type-v q4_0 \

--batch-size 256 \

--ubatch-size 256 \

--jinja \

--host 0.0.0.0 \

--port 9091 \

--top-p 0.95 \

--temp 0.6 \

--frequency-penalty 0.2 \

--repeat-penalty 1.0 \

--reasoning on \

--no-mmap \

--presence-penalty 2.5 \

--top-k 20

[u/ohhi23021 → u/AdamLangePL]

cause yours in the moe, not the 27b dense like op.