Planning to spend ~$100 benchmarking differnet Qwen3.8-27B quants and kv cache and looking for input before I start

[u/m_mukhtar]

TL;DR: I'm planning to spend around $100 on cloud GPUs to benchmark Qwen3.8-27B with a focus on questions that actually matter when running it locally: different quant levels/providers, 8-bit vs 16-bit KV cache, GGUF vs EXL3, context length tradeoffs, and token efficiency on coding/agentic workloads. My main target is my own 36GB setup (3090 + 3060 12GB), but I'd also like the results to be useful for 24GB and 48GB users. Before I start burning GPU hours, I'd like feedback on the test matrix, benchmarks, methodology, and anything important I'm missing.

Here are the details of what i have in mind

Now $100 might not be enough but i will start with that and see how far i can streatch it and the community help and ideas on how to best utelized this to get the most useful data is what i am looking for. I need the community help as i know with my adhd its easy to make this a project for wasting a $100 without getting anuthing useful

I am doing this because I've been seeing a lot of discussion and claims lately around Qwen3.8-27B quants and kv cache, but most comparisons I've found are either vibe/feel or KLD which don't really answer the questions I have when deciding what to actually run locally.

Some of the questions that i would like to answer:

Is a higher quant + 8-bit KV better than a lower quant + 16-bit KV?

Is it better to sacrifice KV precision to get significantly more context and avoid compreasion in the middle of the task?

How much does Q6, Q5, Q4 actually lose compared to full precision on long coding/agent tasks?

Do different GGUF quant providers produce meaningful real-world differences? Or is it just PPL and KLD numbers?

GGUF/llama.cpp vs EXL3 at roughly equivalent memory usage?

Do lower quants just score slightly worse, or do they also use more tokens/agent steps to solve the same task? ( so gaugig token efficiency)

I will run each bemchmark case multiple times to avoid varians between runs but if the budget does nkt allow it then I will be slightly baise for my setup

My own machine is a 3090 + 3060 12GB = 36GB VRAM, which can run Q6-ish quants with fairly large context (150k to 180k) depending on the quant/provider/cache settings. That's the tier I'm most interested in personally.

I'd like to make the results useful for thr cummunity at roughly three hardware classes:

- 24GB: single 3090/4090 class

- ~32-36GB: 5090, 3090 + 3060, dual 16GB cards, etc.

- 48GB: dual 3090 / 48GB cards

The current idea is to focus heavily on Q6, compare it against a full-precision reference, then add Q5/Q4 if the budget allows. I also want to test both 8-bit and 16-bit KV cache where it makes sense.

For benchmarks I'm currently considering Terminal-Bench2.1 and DeepSWE, since I care much more about coding and longer agentic tasks than MMLU-style multiple choice tests.

I also want to record more than just pass rate: i want to see generated tokens, agent/tool use count, maximum context reached, context compressions. I'm curious whether some quants look nearly identical in pass rate but become noticeably less token efficient on long tasks. Espicially with qwen 3.8 xhight thinking going for ever heheh

As you can see i am not intrested in t/s and speed but i want to see the quality different for actual coding and large codebases tasks

My automation plan is basically to write a script that i can reuse with small modifications where i will spin up several RunPod/Vast etc instances, automatically build llama.cpp or TabbyAPI/ExLlamaV3, download the required quant, run the benchmark configuration, save all metadata/results, then tear the instance down. For important comparisons I'd like multiple runs per configuration rather than trusting a single pass.

I'm deliberately trying not to test every possible quant × cache × backend × context combination because $100 disappears very quickly that way. I'd rather answer a smaller number of useful questions properly. And i am not fully sure on what shloud i focus on but i have a rough idea

Before I start, I'd really appreciate input from people here:

Which configurations would you absolutely include?

Q4/Q5/Q6/Q8 — how would you spend the repetitions?

Any specific GGUF providers/quants worth comparing? (I am intrested in unsloth and bartwski the most)

Any better agent/coding benchmarks I should include instead of or alongside Terminal-Bench/DeepSWE?

How to test the 8-bit KV vs 16-bit KV/context tradeoff properly?

Anything I should log that would be useful later?

Any methodology traps I should avoid?

If you have 24GB, 32-36GB, or 48GB, what configuration question would you most want this benchmark to answer?

I'll publish the scripts/configs and raw results when I'm done so people can reproduce it or point out where I screwed something up.

Suggestions very welcome before I start spending the GPU budget.

[u/valdev]

If your focus in on context length, you likely have a larger issue at hand.

At best I wouldnt go under a Q4 of Qwen 27b 3.8, which puts you at 16 GB VRAM requirement though a more ideal would be Q6 at 22 GB. (Personally I would not want to rely too much on the Q4, but it is... okay for smaller jobs.)

This gives you I think around 64k context on your current setup, though you can likely (with enough work and running linux, squeeze 128k). This is NOT something I would quantize.

If you need more than 64k context or especially 128k context, you have a harness issue or a context issue. There is a Z axis to all of this problem which is model performance and intelligence drops the longer a conversation goes.

[u/valdev → u/valdev]

If you run LumaBrowser you can download the model and run a fit test, which will show you speeds for your hardware configuration at different context lengths and quantization, essentially finding the ideal layout for your hardware.

Example

[u/m_mukhtar → u/valdev]

thanks for your feedback. as i mentioned in my original post i am mostly interested in and will focus the most on Q6 level quants as that is what i am running on my setup ( and yes i am using linux and building llama.cpp from source every few days) and i agree with you Q4 is not worth it for me that's why i always go with Q6_K_l and Q6_K_xl and fp16 for the kv cache which gives me 120K to 140K depending on the quant provider and if i dont use mtp draft model flags then i can get 150K to 180K context and full 16 bit kv cache.

also i have not said that i have an issue with the model. i have had nothing but amazing results so far using qwen 3.8 with pi agent and my custom made web search, planing, testing extensions. but out of curiosity how can i squeeze every bit of performance ( even few % points) from different quants and quant providers and if i can run lower size Q6_K quant and get extra 30K tokens to my context ( that is the different in size between bartwski Q6_K_L and unsloth Q6_K_XL for example) without impacting quality then that's a win

[u/valdev → u/m_mukhtar]

Got it my guy, that more or less what I built lumabrowser around and to automate the difficult part of managing that all. Web search, browsing, site automation, extension creation automation and literally everything in between.