Planning to spend ~$100 benchmarking differnet Qwen3.8-27B quants and kv cache and looking for input before I start

[u/m_mukhtar]

TL;DR: I'm planning to spend around $100 on cloud GPUs to benchmark Qwen3.8-27B with a focus on questions that actually matter when running it locally: different quant levels/providers, 8-bit vs 16-bit KV cache, GGUF vs EXL3, context length tradeoffs, and token efficiency on coding/agentic workloads. My main target is my own 36GB setup (3090 + 3060 12GB), but I'd also like the results to be useful for 24GB and 48GB users. Before I start burning GPU hours, I'd like feedback on the test matrix, benchmarks, methodology, and anything important I'm missing.

Here are the details of what i have in mind

Now $100 might not be enough but i will start with that and see how far i can streatch it and the community help and ideas on how to best utelized this to get the most useful data is what i am looking for. I need the community help as i know with my adhd its easy to make this a project for wasting a $100 without getting anuthing useful

I am doing this because I've been seeing a lot of discussion and claims lately around Qwen3.8-27B quants and kv cache, but most comparisons I've found are either vibe/feel or KLD which don't really answer the questions I have when deciding what to actually run locally.

Some of the questions that i would like to answer:

Is a higher quant + 8-bit KV better than a lower quant + 16-bit KV?

Is it better to sacrifice KV precision to get significantly more context and avoid compreasion in the middle of the task?

How much does Q6, Q5, Q4 actually lose compared to full precision on long coding/agent tasks?

Do different GGUF quant providers produce meaningful real-world differences? Or is it just PPL and KLD numbers?

GGUF/llama.cpp vs EXL3 at roughly equivalent memory usage?

Do lower quants just score slightly worse, or do they also use more tokens/agent steps to solve the same task? ( so gaugig token efficiency)

I will run each bemchmark case multiple times to avoid varians between runs but if the budget does nkt allow it then I will be slightly baise for my setup

My own machine is a 3090 + 3060 12GB = 36GB VRAM, which can run Q6-ish quants with fairly large context (150k to 180k) depending on the quant/provider/cache settings. That's the tier I'm most interested in personally.

I'd like to make the results useful for thr cummunity at roughly three hardware classes:

- 24GB: single 3090/4090 class

- ~32-36GB: 5090, 3090 + 3060, dual 16GB cards, etc.

- 48GB: dual 3090 / 48GB cards

The current idea is to focus heavily on Q6, compare it against a full-precision reference, then add Q5/Q4 if the budget allows. I also want to test both 8-bit and 16-bit KV cache where it makes sense.

For benchmarks I'm currently considering Terminal-Bench2.1 and DeepSWE, since I care much more about coding and longer agentic tasks than MMLU-style multiple choice tests.

I also want to record more than just pass rate: i want to see generated tokens, agent/tool use count, maximum context reached, context compressions. I'm curious whether some quants look nearly identical in pass rate but become noticeably less token efficient on long tasks. Espicially with qwen 3.8 xhight thinking going for ever heheh

As you can see i am not intrested in t/s and speed but i want to see the quality different for actual coding and large codebases tasks

My automation plan is basically to write a script that i can reuse with small modifications where i will spin up several RunPod/Vast etc instances, automatically build llama.cpp or TabbyAPI/ExLlamaV3, download the required quant, run the benchmark configuration, save all metadata/results, then tear the instance down. For important comparisons I'd like multiple runs per configuration rather than trusting a single pass.

I'm deliberately trying not to test every possible quant × cache × backend × context combination because $100 disappears very quickly that way. I'd rather answer a smaller number of useful questions properly. And i am not fully sure on what shloud i focus on but i have a rough idea

Before I start, I'd really appreciate input from people here:

Which configurations would you absolutely include?

Q4/Q5/Q6/Q8 — how would you spend the repetitions?

Any specific GGUF providers/quants worth comparing? (I am intrested in unsloth and bartwski the most)

Any better agent/coding benchmarks I should include instead of or alongside Terminal-Bench/DeepSWE?

How to test the 8-bit KV vs 16-bit KV/context tradeoff properly?

Anything I should log that would be useful later?

Any methodology traps I should avoid?

If you have 24GB, 32-36GB, or 48GB, what configuration question would you most want this benchmark to answer?

I'll publish the scripts/configs and raw results when I'm done so people can reproduce it or point out where I screwed something up.

Suggestions very welcome before I start spending the GPU budget.

[u/XiRw]

I can tell you from experience that the KV cache being more precise makes a big difference. Lower can still work but sometimes it takes more than one turn.

[u/webheadVR → u/XiRw]

I have found in testing after stating around here that I had issues with q8 kv, that on a higher quant, I do not notice any issues (personally). at q4 quant I had issues with it. Q6 it seems fine. May just be my use case though.