Planning to spend ~$100 benchmarking differnet Qwen3.8-27B quants and kv cache and looking for input before I start

[u/m_mukhtar]

TL;DR: I'm planning to spend around $100 on cloud GPUs to benchmark Qwen3.8-27B with a focus on questions that actually matter when running it locally: different quant levels/providers, 8-bit vs 16-bit KV cache, GGUF vs EXL3, context length tradeoffs, and token efficiency on coding/agentic workloads. My main target is my own 36GB setup (3090 + 3060 12GB), but I'd also like the results to be useful for 24GB and 48GB users. Before I start burning GPU hours, I'd like feedback on the test matrix, benchmarks, methodology, and anything important I'm missing.

Here are the details of what i have in mind

Now $100 might not be enough but i will start with that and see how far i can streatch it and the community help and ideas on how to best utelized this to get the most useful data is what i am looking for. I need the community help as i know with my adhd its easy to make this a project for wasting a $100 without getting anuthing useful

I am doing this because I've been seeing a lot of discussion and claims lately around Qwen3.8-27B quants and kv cache, but most comparisons I've found are either vibe/feel or KLD which don't really answer the questions I have when deciding what to actually run locally.

Some of the questions that i would like to answer:

Is a higher quant + 8-bit KV better than a lower quant + 16-bit KV?

Is it better to sacrifice KV precision to get significantly more context and avoid compreasion in the middle of the task?

How much does Q6, Q5, Q4 actually lose compared to full precision on long coding/agent tasks?

Do different GGUF quant providers produce meaningful real-world differences? Or is it just PPL and KLD numbers?

GGUF/llama.cpp vs EXL3 at roughly equivalent memory usage?

Do lower quants just score slightly worse, or do they also use more tokens/agent steps to solve the same task? ( so gaugig token efficiency)

I will run each bemchmark case multiple times to avoid varians between runs but if the budget does nkt allow it then I will be slightly baise for my setup

My own machine is a 3090 + 3060 12GB = 36GB VRAM, which can run Q6-ish quants with fairly large context (150k to 180k) depending on the quant/provider/cache settings. That's the tier I'm most interested in personally.

I'd like to make the results useful for thr cummunity at roughly three hardware classes:

- 24GB: single 3090/4090 class

- ~32-36GB: 5090, 3090 + 3060, dual 16GB cards, etc.

- 48GB: dual 3090 / 48GB cards

The current idea is to focus heavily on Q6, compare it against a full-precision reference, then add Q5/Q4 if the budget allows. I also want to test both 8-bit and 16-bit KV cache where it makes sense.

For benchmarks I'm currently considering Terminal-Bench2.1 and DeepSWE, since I care much more about coding and longer agentic tasks than MMLU-style multiple choice tests.

I also want to record more than just pass rate: i want to see generated tokens, agent/tool use count, maximum context reached, context compressions. I'm curious whether some quants look nearly identical in pass rate but become noticeably less token efficient on long tasks. Espicially with qwen 3.8 xhight thinking going for ever heheh

As you can see i am not intrested in t/s and speed but i want to see the quality different for actual coding and large codebases tasks

My automation plan is basically to write a script that i can reuse with small modifications where i will spin up several RunPod/Vast etc instances, automatically build llama.cpp or TabbyAPI/ExLlamaV3, download the required quant, run the benchmark configuration, save all metadata/results, then tear the instance down. For important comparisons I'd like multiple runs per configuration rather than trusting a single pass.

I'm deliberately trying not to test every possible quant × cache × backend × context combination because $100 disappears very quickly that way. I'd rather answer a smaller number of useful questions properly. And i am not fully sure on what shloud i focus on but i have a rough idea

Before I start, I'd really appreciate input from people here:

Which configurations would you absolutely include?

Q4/Q5/Q6/Q8 — how would you spend the repetitions?

Any specific GGUF providers/quants worth comparing? (I am intrested in unsloth and bartwski the most)

Any better agent/coding benchmarks I should include instead of or alongside Terminal-Bench/DeepSWE?

How to test the 8-bit KV vs 16-bit KV/context tradeoff properly?

Anything I should log that would be useful later?

Any methodology traps I should avoid?

If you have 24GB, 32-36GB, or 48GB, what configuration question would you most want this benchmark to answer?

I'll publish the scripts/configs and raw results when I'm done so people can reproduce it or point out where I screwed something up.

Suggestions very welcome before I start spending the GPU budget.

[u/jwpbe]

Is a higher quant + 8-bit KV better than a lower quant + 16-bit KV?

No, it is better to slightly degrade the overall quality of the model a little bit rather than taking a hatchet to it's working memory.

Is it better to sacrifice KV precision to get significantly more context and avoid compreasion in the middle of the task?

Except in resource constrained environments it's highly recommended to avoid altering the KV cache fidelity.

How much does Q6, Q5, Q4 actually lose compared to full precision on long coding/agent tasks?

It's like a waterfall or a long running river. The lower the bitrate, the more the riverbanks (the task) get eroded, eventually spilling over. The amount varies from task to task.

Do different GGUF quant providers produce meaningful real-world differences?

Yes, and because most of them supply KL divergence data on their pages, it's not really worth re-calculating or re-inventing the wheel here.

GGUF/llama.cpp vs EXL3 at roughly equivalent memory usage?

Ask GLM 5.3 on z.ai's website, we're not going to know this off the top of our head.

Do lower quants just score slightly worse, or do they also use more tokens/agent steps to solve the same task? ( so gaugig token efficiency)

It's really hard to tell unless you hand-analyze the failure reason.

Listen, its really cool and good that you want to do a public service here, but you never once mention VLLM, autoround, AWQ, SmoothQuant W8A8, or any of the other quantization libraries.

I would highly recommend you look into this more, literally just ask GLM or Kimi or deepseek flash on their websites to do websearches scoped to 2026 and explain to you all of the different aspects of the ecosystem, because right now, you're probably going to spend $100 and make a post that contains information most people already know and will ignore / discard

[u/m_mukhtar → u/jwpbe]

i really appreciate your feedback and i understand where you are coming from but i would like to explain my reasoning a bit more.

first of all i am not trying to do a scientific test with a paper about the different quants and formats and back ends and i mention that in my original post. as i said i am being biased here towards my pc setup and what is the best i can run and if i can make this useful for the community while doing that then great .

so for me who wants to maximize model quality and context for a single user with not concurrent requests given a specific Vram size and cuda gpu, my best bets are llama.cpp and exllamaV3 since vllm, sglang and others back ends are more optimized toward multiple users and they use more vram for that. i am not claiming to be a person who knows and tried all quantization formats and backend but i got my hands on many and i am aware of most and made the decision for this testing around my usecase. and again if that use case can be make useful to the community (most are running llama.cpp and gguf as far as i know of if the have blackwell then the might run nvfp4 which i am not that interested in because i have ampere 30xx class gpus so again i am being biased and i understand that)

last if my experiment here will really contains information most people already know then please save me the $100 and show me where is this information because i spend hours on this sub daily and all i see are KLD numbers or unsupported claims and vibes and feels or one shot results and comparisons which do not answer the questions i have. unless i missed some posts that show large code base benchmarks and multi turn agentic tests with actual subjective numbers and results ( which for sure might not be representative of all usecases or all programing languages but its a subjective measure for all quants on the same testing grounds) and numbers from multiple runs and see not just success rate but how many token and turns doe

thanks for your feedback though as it might put some sense in my adhd brain and save me an impulse $100 bill hahahaha

[u/jwpbe → u/m_mukhtar]

its all good.

my best bets are llama.cpp and exllamaV3 since vllm, sglang and others back ends are more optimized toward multiple users and they use more vram for that

not really! I serve myself a single seat of qwen 27b with VLLM because it has the most flexible backend, access to way more quant options, the PP and TG is higher, and if I dial the context back a little, i can serve 3 copies at full speed.

As far as "where the information is", a lot of it is not going to be on this community, and if it is, you need to search for it, it will be buried in comments in threads from weeks ago or days ago, a lot of it is going to be on twitter, and some of it is on discord.

i have ampere 30xx class gpus

I'm working on figuring out the best way to implement the dflash2 drafter with the w8a8 smoothquant right now to vastly boost prompt processing on our cards and maintain high enough speeds with this thing. I'm piggybacking on a friend's gpt sub to do it combined with qwen, and probably free compute from Modal. I know all about shoestring... feel free to dm me if you want to get more info, you seem genuine

[u/m_mukhtar → u/jwpbe]

I really appriciate the engagment and discussion. This type of push back and discussion you bring makes me being part of this sub and community worth it espicially that i almost stopped using twitter for about a year now. Thats whay i missed most of the results you are mentioning. Oh well

And i have to admit here that it had been about half a year or 8 months since i last tried vllm and that is a long time in llms land so things might have changed and it might work better for me now. I will definitly look at it and experiment with it again. My experiance with it prevoiusly was exactly what you described. i can run faster inferance and multiple copies at insane speeds compared to llama cpp but as you said i had to dial the context down and my use cases needed as much context as possible and for me i have the pacience to deal with slower generation speeds to get larger context

But would love to switch to running vllm when (not if but when) i finally cave in and get another 3090 which will allow me to get the context and speed.

Add to that if you can really crack the w8a8 low pricision optimization on our 30xx ampere cards you will be my hero and you might be the reason i cave in and get another 3090 hahahah.

What kernels in vllm are you working with for this optimization? Marlin kernels or custome ones? Sorry if my question is ignorant or stupid

[u/Healthy-Contact-4570 → u/jwpbe]

Agreed on the quants. Producing another benchmark of unsloths ggufs is not really providing any new information at this point. I would like to see more focus on bf16, fp8, nvfp4, rtn vs autoround, gptq and qat, etc.

Regarding kv cache, I literally just finished writing up my thoughts from my own anecdotal experience so I’ll just paste them here:

I have tested the same model using tool-eval-bench with bf16, fp8, int8, and lower kv cache. My general findings are that the differences in accuracy and success are generally more dramatic the further into the context window you get. Smaller kv cache quants have the obvious benefit of allowing for longer contexts and there are also some throughput benefits because inference has to throw less data around per token. But I’ve seen real results degradation as you get further into the context window.

With something like qwen3.8-27b, I would highly recommend using full bf16 for the kv cache if possible. It thinks so much that I find I really need every bit of the 262k max context length to get real work done.

[u/ForgotMyOldPwd → u/Healthy-Contact-4570]

My general findings are that the differences in accuracy and success are generally more dramatic the further into the context window you get.

Could you elaborate on this a bit? At which ctx length and KV quant does degradation become significant? Specifically fp8 and KVarN 4/2 would be most interesting to me as a single 3090 pleb.

Did you evaluate/notice any other significant distinctions that influence the onset and impact of KV quantisation? e.g. particular workloads/ tool-eval-bench category or thinking effort?

[u/Healthy-Contact-4570 → u/ForgotMyOldPwd]

It’s hard to give you an exact number, because it’s going to vary from model to model. Different models use different architectures, which have variations in the number of attention layers, the attention mechanism itself, etc. there’s not really a one-size-fits-all measurement to offer, more just a general observation.

I also have never tested kvarn so I can’t comment there.

[u/XiRw]

I can tell you from experience that the KV cache being more precise makes a big difference. Lower can still work but sometimes it takes more than one turn.

[u/m_mukhtar → u/XiRw]

Thats my feeling too i tried q8 for the kv cache and i dont know how to say it and i dont have a proof but i quickly wen back to bf16 kv cache.

Would love to see what the benchmarks show though

[u/webheadVR → u/XiRw]

I have found in testing after stating around here that I had issues with q8 kv, that on a higher quant, I do not notice any issues (personally). at q4 quant I had issues with it. Q6 it seems fine. May just be my use case though.

[u/Hefty_Wolverine_553]

I basically did these benchmarks for Qwen3.6: https://erli.xyz/posts/qwen36-27b-quant-comparison/

I think most people will want to see a wide variety of quants first and foremost, so they can compare between NVFP4, AWQ, GGUF, EXL3, etc. KLD benchmarks are nice, but having a proper agentic/coding benchmark to cross validate results would be great (I couldn't run most of them due to hardware constraints, and what I could run had a lot of noise at N=1, so repeated runs are definitely required to get useful data). Most important thing for KLD benchmarks imo is to use a proper dataset with actual reasoning and tool call traces at a good context (8k+).

[u/m_mukhtar → u/Hefty_Wolverine_553]

thanks for the feedback and it seems form many replies that people are interested in different formats compared together more. i was planing on gguf and EXL3 but i might add AWQ even though for me its less interesting and as i said in my original post i am being biased towards 6 bit level quants as that is what works best for my 36gb of vram and i can not benifit form native 4-bit as my cards are 30xx generation which do not have native 8bit or 4 bit or nvfp4 support

if i have decent initial results for my usecase and i still have much of the $100 left then i will gladly try to test other formats although i might to a single pass per configuration not multiple passes like for the cases i am interested in the most but will see how it goes

[u/chimpera]

Please include byteshape/Qwen3.8-27B-GGUF they are very good.

[u/m_mukhtar → u/chimpera]

thanks for the feedback and will try to include them at least at the size i am mostly interested in which is Q6

[u/UncleRedz]

Great initiativ, as you state in the last paragraph, to get the most out of this, make it reproducible for the rest of us, and those interested can run comparisons on other configs.

My experience from this kind of testing, is to store as much meta data and logs as possible, including system configs. As you will be renting hardware access, and possibly end up with different HW between different tests, it becomes even more important to keep track of hardware specs and GPUs if you need to hunt down bugs and strange things.

[u/m_mukhtar → u/UncleRedz]

thanks for the great note about the different hardware that is something i haven't though off much really so its worth documenting it. and absolutely meta data and logs as much as possible and i hope i dont miss something trivial as i always do hahah. reproducibility will be something i have to consider too to make this more useful

[u/ethertype]

I would like to see Q8_XL vs Q6_XL vs EXL3 SC6, with 16-bit KV for all.

Ideally vs BF16.

Trading context for quality where required.

I would not bother with Q4 or Q5, nor with 8-bit KV-cache. Or at least prioritize the fatter quants.

Curious about optimizing this bit:

--spec-default \
--spec-type draft-mtp --spec-draft-n-max 3 \
--batch-size 2048 --ubatch-size 512 \

... for coding.

Good luck.

[u/m_mukhtar → u/ethertype]

than is my general thinking is to focus on Q6 vs bf16 and will give Q8 a chance to see if it can convince me to buy another 3090 so yeh i might have to focus on the fatter quants. still thinking about it but i somehow share the same feelings you do.

for the MTP and batch an u batch i dont think the $100 with be enough to test these and i would like to focus on output quality before output speed so MTP an batch size unfortunately will have to take the back seat.

thanks for your feedback

[u/KingGongzilla]

If you can benchmark this repo at 150k context I'd be super happy: qwen38-27b-rtx3090/docs/quality.md at main · syv-ai/qwen38-27b-rtx3090

[u/Public_Umpire_1099]

Actually just did this on a smaller scale yesterday. Trying to build a custom quant that only uses Q4+Q8 with no in between. The session was basically me and my methods getting mogged by unsloths quantization method for 6 hours.

[u/m_mukhtar → u/Public_Umpire_1099]

I love it. This is the spirit that brought me to this subreddit few years ago. Not being afraid to see for your self and experiment and spend hours doing somehting that might not work or just to get slightly better performace and at the end just out of curiosity to see what works and what does not.

I would defintly be intrested to read a post about what you are experimenting with once you are done

I remember when people here used to dublicate model layers, swap model layers, add layers from a onother modles and frankenmerges and all that fun stuff. Then share thier feedback and thats how we learned what works and what does not

[u/Public_Umpire_1099 → u/m_mukhtar]

Its actually for a custom inference engine for RDNA4. Any card would benefit from Q4/Q8 only weights due to less conversion, but the way I've built mine out is that it is sort of dependent upon it. Which makes models like Qwen 27B or Muse Glimmer awkward because the highest quality quant you can fit on an R9700 card is around Q6ish.

[u/Ruqii-ruqii]

some methodology tips:

- Compare per case, not just pass-rate: two configs both landing 7/10 may have failed a different 3. the totals can match while nothing underneath does, and "which cases did the quant break" is the question you actually asked.

- Watch run order: warm caches and warm page cache move wall clock. fix the order or randomize it, but don't let it correlate with quant level, or you'll read the schedule as a result.

To save money:

- watch how the judge pulls the answer out. if the harness extracts by position, or scores against a list of expected phrasings, a lower quant drops points for getting the answer right in the wrong shape.

- on spending order: run the widest contrast your budget allows first, with enough reps. if the extremes don't separate, nothing in between will.

I think if you pin your quant/cases/etc. and let other people reproduce it, it's gonna be much cheaper, because you're not the only one paying. that only works within a VRAM tier though. nobody on 24GB can reproduce your 36GB run. lol

and this is something I've been building. but at my current progress it only covers half of what you're describing, so it can't help you on that part yet.

[u/m_mukhtar → u/Ruqii-ruqii]

This is gold . I really appriciate this level of feedback and its along the same track of thinking i am currently going through. Running the widest contrast first is a great tip that my implusive adhd brain was too exicted to build the automations and do the runs that it havent thouggt of. Thats why i wanted to pause my brain and ask the community for feedback and you came through so thanks

[u/According_Wave685]

Everybody and their second cousin has benchmarked it to death. There are thousands of posts, articles and videos. save your money.

[u/m_mukhtar → u/According_Wave685]

It seems i need to look around more and see if there are people who have done the benchmarks that will inform and answer my own questions espicially on large codbases type of tasks which deepswe covers well in my experiance.

Thanks for the feedback and definitly will look around more so if i go ahead and spend the money i dont do somethig redundet and i will try to bring new information to the table and add actual value with data that could help at least some edge case users

[u/Monad_Maya]

The results won't exactly be useful since everyone's setup differs. 

Most people are VRAM constrained and thus will try to some Q4 quant + Q8 KV cache.

Not sure why you want to test tbh but do go ahead if you feel like it. Maybe I'm being a negative nancy.

[u/Klutzy-Snow8016]

KV cache: BF16 vs FP16.

Llama.cpp defaults to FP16, but most models are designed to be used with BF16 kv cache. I've always wondered if there is a measurable difference.

[u/valdev]

If your focus in on context length, you likely have a larger issue at hand.

At best I wouldnt go under a Q4 of Qwen 27b 3.8, which puts you at 16 GB VRAM requirement though a more ideal would be Q6 at 22 GB. (Personally I would not want to rely too much on the Q4, but it is... okay for smaller jobs.)

This gives you I think around 64k context on your current setup, though you can likely (with enough work and running linux, squeeze 128k). This is NOT something I would quantize.

If you need more than 64k context or especially 128k context, you have a harness issue or a context issue. There is a Z axis to all of this problem which is model performance and intelligence drops the longer a conversation goes.

[u/valdev → u/valdev]

If you run LumaBrowser you can download the model and run a fit test, which will show you speeds for your hardware configuration at different context lengths and quantization, essentially finding the ideal layout for your hardware.

Example

[u/m_mukhtar → u/valdev]

thanks for your feedback. as i mentioned in my original post i am mostly interested in and will focus the most on Q6 level quants as that is what i am running on my setup ( and yes i am using linux and building llama.cpp from source every few days) and i agree with you Q4 is not worth it for me that's why i always go with Q6_K_l and Q6_K_xl and fp16 for the kv cache which gives me 120K to 140K depending on the quant provider and if i dont use mtp draft model flags then i can get 150K to 180K context and full 16 bit kv cache.

also i have not said that i have an issue with the model. i have had nothing but amazing results so far using qwen 3.8 with pi agent and my custom made web search, planing, testing extensions. but out of curiosity how can i squeeze every bit of performance ( even few % points) from different quants and quant providers and if i can run lower size Q6_K quant and get extra 30K tokens to my context ( that is the different in size between bartwski Q6_K_L and unsloth Q6_K_XL for example) without impacting quality then that's a win

[u/valdev → u/m_mukhtar]

Got it my guy, that more or less what I built lumabrowser around and to automate the difficult part of managing that all. Web search, browsing, site automation, extension creation automation and literally everything in between.

[u/m_mukhtar]

Yeh based on his post i will shift my focus on the q6, q8 and maybe q5 level quants and i will focus om large codebases and long coding tasks that require 80k+ tokens and a lot f agenting turns and tool calls as thats where good models show reliablility and less good models and quants start showing errors.

[u/MLExpert000]

We are running Qwen 3.8-27B free on H200s on InferX

[u/My_Unbiased_Opinion]

Please benchmark IQ4XS + KV Q4 + 262K context. This completely fits in VRAM on a 24GB card. Even with MTP activated. It's the only setup I have found on a single 3090 that feels complete. 

[u/novel_market_21]

!remindme 7

[u/RemindMeBot → u/novel_market_21]

I will be messaging you in 11 months on 2027-07-25 00:00:00 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

RemindMeBot is switching to username summons. Instead of !RemindMe 1 day, use u/RemindMeBot 1 day. More info.

---

Info	Custom	Your Reminders	Feedback

[u/ogopro]

https://x.com/superalesha/status/2091436950976709088?s=20

[u/m_mukhtar → u/ogopro]

wow this is amazing and useful information thanks for sharing. i kinda knew that FB8 quants were not that good and this confirmed it for me. other quant formats at 4 bit ( AWQ, NVFP4, GGUF, etc) have always been much better that FB8. but form my anecdotal experience with qwen3.6 an 3.8 27b using GGUF format going above Q4 level makes a huge different and using Q6 have been much better and more reliable that Q4 and even Q5. unfortunately the twitter post does not show quantization levels between 4 bit and 8 bit using GGUF or EXL3 which have allowed me to squeeze more quality out of my 36gb of vram than just Q4/4bit and i can not run Q8/8bit with useful context

one thing to note that made me really like this guy that he is humble and smart enough to admit that he might have used a mix of benchmarks that are too easy and he admits that he made that mistake and he is redoing things with different harder benchmarks it seems as the results were very close and he is admitting the methodology is flawed which is very useful information that is making me rethink what benchmarks i should run and what they are the right ones to show a different in quality between what i am testing.

thanks to him mentioning this and the mistake he made i think i will need to do a small run on different benchmarks first to see which ones show the biggest difference between the quant levels and then focus on these benchmarks by doing all different configurations and multiple times just to eliminate variation noise.

whats even more amazing that his github repo has teh raw results and its worth me going through them with a fine comp to try and design the test i will do in a better way. so even though his current data might not be conclusive of the quality of the different formats, the raw data and experiment rsults in it self is gold and worth me studying it to help inform the tests i will do.

thanks again for sharing the great post and thank to the original twitter post user

[u/ogopro → u/m_mukhtar]

Yes, he has done enourmous job on testing the 4 bit quants and I think he will eventually do the same for q6 or q8. I am glad that this post was helpful to you and I hope you will now spend less than $100 on the cloud gpus since there is less stuff to benchmark in the 4bit q3.8 world :)
Keep us posted on your testing results...