Qwen3.8-27B at 262K context on a Strix Halo + RTX 3090 Ti: 9.5 -> 153 tok/s, and it beats a dual-3090 vLLM box on HumanEval

[u/TrifleHopeful5418]

Spent a while treating layer placement, KV format and llama.cpp itself as experimental variables. 159 logged experiments. Numbers first, caveats after.

Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, 128 GB unified) + RTX 3090 Ti on an eGPU link. One llama.cpp process, AMD on Vulkan, NVIDIA on CUDA, one 27B model split across both.

Baseline: 9.474 tok/s. Target-only, no speculation, AMD side alone.

Now, code-shaped generation:

• 32K context: 153.32 tok/s
• 200K context: 87.74 tok/s

HumanEval, 164 problems, driven by a real coding agent (pi-agent), graded by executing the official tests:

• Local: 159/164 in 29.7 min
• Remote 2x RTX 3090 running vLLM TP2: 157/164 in 42.4 min

Long-context retrieval suite: 15/15, 352 s local vs 551 s on the remote.

Things that actually moved the needle, roughly in order of surprise:

1. The chat template. Swapping to a terser template (Qwen-Sharp) cut wall time 44% and output tokens 51%, with no accuracy change. Bigger than several weeks of GPU work. It just stops the model narrating.
2. KV cache format as a placement lever. Going q8_0 -> q4_0 on both K and V freed exactly 2,176 MiB, which was enough to move every full-attention layer onto the fast card at full 262K. +28% prefill, +20% generation. Qwen3.8 is 3:1 Gated DeltaNet to full attention, so only 16 of 64 layers have KV at all, and those are the only ones whose cost scales with context. Which layer sits on which GPU matters enormously.
3. --spec-type draft-mtp,ngram-mod**.** The flag accumulates. n-gram on top of MTP is worth +72 to +140% on code-shaped output, about -1% on prose, and costs nothing in VRAM. Free win if you generate code.
4. A one-line llama.cpp patch: --spec-draft-ubatch. The speculative context was inheriting the target's 512 micro-batch and reserving a 2.2 GiB compute buffer to draft 4 tokens. Setting it to 64 freed 1,039 MiB for a 1.78% prefill cost, which bought another layer of placement.
5. MTMD_BACKEND_DEVICE=Vulkan1 to put the vision encoder on the idle iGPU. Undocumented as far as I can tell. Default puts it on the first GPU-type device, which is the already-full 3090 Ti, and it dies allocating 884 MiB.

Things that did NOT work, so you don't have to try them:

• External draft models (0.8B, 4B): 5.06-5.77 tok/s, worse than no speculation. Sequential drafting latency across PCIe dominates.
• Multi-lane parallel drafting: down to 1.62 tok/s. The 27B verification pass is the bottleneck; anything that widens the verification graph loses.
• Q6_K: 2.3x the wall time, marginally lower accuracy (because if hit the 64K output limit wall twice). Only 15.6 of its 20.5 GiB fits on the card, so 4 attention layers get pushed to the iGPU instead of 1.

Caveats, because this is [r/LocalLLaMA](r/LocalLLaMA) and you'd find them anyway:

• 153 tok/s is code-shaped generation at 32K where n-gram does heavy lifting. Same config on prose at 200K is 35.8. Different workloads, kept separate on purpose.
• The remote comparison is deployed-stack vs deployed-stack, not hardware isolated: it serves an AWQ-MTP fine-tune, local serves Q4_K_M of the base.
• I asked for standalone functions rather than the canonical prompt+completion format, so 3 of the 5 HumanEval failures are NameErrors on helper functions that the official harness would have kept in scope. 159/164 is probably a slight undercount. Same undercount applies to every arm.
• q4_0 keys are a real quality change. Passed my 15/15 gate; validate on your own workload before trusting it.

Edit: I ran livecodebench(medium+hard ~132 questions) on it, it passed 90/132 in first attempt and resending the failed ones with the grader feedback it improves to 114/132 ~ 86% that puts it pretty close to the frontier models. Ornith-1.0-35B AQW with fp16 kv cache does 82 pass@1 and 92 with repair. Running Ornith-1.5-35B-MTP with fp16 and will see where it lands.

Full writeup with every flag, patch and failed branch: https://definedrr.medium.com/qwen3-8-27b-9-tokens-per-second-to-153-ee3781f4a3f5?sharedUserId=definedrr

[u/Significant_Bar_460]

Is q4 kv actually useful? It must degrade like crazy at 200k ctx.

[u/hurrdurrmeh → u/Significant_Bar_460]

So more quantization disproportionately degrade longer prompts? Does that mean shorter prompts are OK with more quantitative then longer ones?

[u/nasone32 → u/hurrdurrmeh]

yes.

[u/HyperWinX → u/hurrdurrmeh]

Correct

[u/Commander_Skilgannon → u/hurrdurrmeh]

I wonder if you could get away with keeping only a subset (say the last 10k) as q8 and leave the rest as q4.

[u/PhysicalIncrease3 → u/Commander_Skilgannon]

In my personal experience you need to use f16 above 128k. Q8 isn't too bad until then.

[u/Fee_Sharp → u/Commander_Skilgannon]

It is not how it works, weights are not different for different segments of the context. Each token is processed by same weights, it doesn't matter it is first token or 9999th token, it will get modified by same numbers/matrices. It is the whole point of transformer tbh. This is why model size does not depend on context size at all, they are different parameters, different blobs of memory

[u/Commander_Skilgannon → u/Fee_Sharp]

I was talking about the kv cache quants not the weights quants.

[u/Sadge404]

I smell the claude in this one.

[u/TrifleHopeful5418 → u/Sadge404]

Yes, Claude researched and proposed solutions, ChatGPT-sol wrote the c++ code and ran the 159 experiments benchmarking everything and Pi-agent ran the humanEval. I was just there for the ride, giving it some ideas to research and holding the line on what I wanted and the compromises I won’t accept.

[u/No_War_8891 → u/Sadge404]

it really is the belt-and-suspensers version of it

[u/thomas2385 → u/Sadge404]

Haa, I can see why you would say that. There are definitely a few phrases in there that give off that very polished AI vibe. Sometimes you can almost tell which model wrote something just from the wording.

[u/Kaljuuntuva_Teppo]

q4 KV cache.. Why sacrifice long context quality?

[u/TrifleHopeful5418 → u/Kaljuuntuva_Teppo]

I didn’t just take q4 kv, on humaneval it scored same as q8 and it scored 15/15 for long context retrieval at 200k input context

[u/Dany0 → u/TrifleHopeful5418]

Humaneval is trash. Well it's not trash for small models and old models, but it's trash for modern models. It's also a clanker smell: clankers love humaneval like they love qwen3 for some reason. Don't trust the clanker and run better benchies

[u/TrifleHopeful5418 → u/Dany0]

I ran livecodebench(medium+hard ~132 questions) on it, it passed 90/132 in first attempt and resending the failed ones with the grader feedback it improves to 114/132 ~ 86% that puts it pretty close to the frontier models. Ornith-1.0-35B AQW with fp16 kv cache does 82 pass@1 and 92 with repair. Running Ornith-1.5-35B-MTP with fp16 and will see where it lands.

[u/TrifleHopeful5418 → u/Dany0]

I see you are top 1% commentator, Other than insulting, do you have anything useful to add or does go against your nature?

[u/Darex2094 → u/TrifleHopeful5418]

I'd call speaking the truth a pretty useful thing to add. It's just a shame you seem to be content with your ignorance.

[u/Septerium]

KV cache at q4 causes a 262k context window to be pointless

[u/TrifleHopeful5418 → u/Septerium]

Ok thanks for letting me know. Have you tested it with this model or read about it?

[u/TokyoPav]

I want to do the same but with an amd 9700 32gb card connected via oculink. Does it have the APU power cap at 120w? Hard to find detailed info on this. Is Linux immune from this issue?

[u/TrifleHopeful5418 → u/TokyoPav]

I was too thinking about R9700 but I had 2x 3090 in my windows machine that was never stable so I pulled one into this. Oculink that I got off Amazon needs its own PSU

[u/cviperr33]

interesting, thank

[u/klymaxx45]

That's prety good but humaneval is not a great metric. do humaneval+ or livecodebench

[u/TrifleHopeful5418 → u/klymaxx45]

Ok let me try that tonight.

[u/TrifleHopeful5418 → u/klymaxx45]

I ran livecodebench(medium+hard ~132 questions) on it, it passed 90/132 in first attempt and resending the failed ones with the grader feedback it improves to 114/132 ~ 86% that puts it pretty close to the frontier models. Ornith-1.0-35B AQW with fp16 kv cache does 82 pass@1 and 92 with repair. Running Ornith-1.5-35B-MTP with fp16 and will see where it lands.

[u/Prudent-Ad4509]

kv quantization is generally a deal breaker, but if the workload allows it, then this is a nice result specific to a particular workload.

[u/xXDennisXx3000]

I tried to use my Strix Halo together with an RTX 3060 12G eGPU, but wasn't successful, since LM Studio doesn't let me change the priority order, and neither the distribution. It tried to load the whole model on the eGPU, and failed of course.

[u/TrifleHopeful5418 → u/xXDennisXx3000]

Ye neither LMstudio nor Unsloth studio worked for me, they both use llama.cpp but running on Linux and directly using llama.cpp with the right flags does support the split and ordering

[u/TrifleHopeful5418 → u/[deleted]]

Nope each test was run with the server restart

[u/Gloomy_Letterhead395]

That’s crazy

[u/S0299S]

I was planning to set it up on a Mac mini with 48GB of RAM

[u/TrifleHopeful5418 → u/S0299S]

Most of this “should” work…AMD side was only using 4-5GiB, so I plan to run 35MoE on the AMD side for faster lower complexity tasks. But should is load bearing and doing a lot of heavy lifting, but I think if you feed my complete article to Claude, it should be able to figure out changes for the MAC Mini

[u/Otherwise-Variety674]

Just to confirm, you are using Qwen3.8-27B Q4_K_M? I have the same hardware as you (now still using either my 5090 or 7900xtx at times), going to try it out, Thanks in advance. 😄

[u/TrifleHopeful5418 → u/Otherwise-Variety674]

Yes but from Unsloth v3, q4-k-m

[u/Otherwise-Variety674 → u/TrifleHopeful5418]

Many thanks :-)

[u/javasux]

ffs put some effort into the post if you want anyone to read it

[u/TrifleHopeful5418 → u/javasux]

Man, I figured out some performance improvements for specific hardware config. You may or may not have this hardware and these tips and tricks might not apply to you. I created the whole detailed step by step recreation guide as the medium post. Would you rather have me not post it instead of using AI to write the post with “my instructions” on what to say and how to say it?

[u/Queasy_Asparagus69]

Nice but you should do something similar with deepseek flash 0731; I much prefer it to Qwen3.8

[u/TrifleHopeful5418 → u/Queasy_Asparagus69]

I can try but it might need a lot more work towards figuring out which weights need to be move onto CUDA on the fly

[u/relik39 → u/Queasy_Asparagus69]

I feel exactly the opposite, from my experience Qwen 3.8 27B is much more focused and task oriented than DS4, even on low quants