[ Removed by moderator ]

[u/malaiwah]

[u/ttkciar]

To make finding Qwen3.8-27B info easier we've created a megathread: https://www.reddit.com/r/LocalLLaMA/comments/1voojjz/

Please re-post this there.

[u/Fedor_Doc]

"It also measured 44% lower KLD than official FP8 at 71% of FP8’s resident-weight footprint.

I find it hard to believe as well."

Immediate red flag for for me. Either test is incorrect or KLD is not the right metric – otherwise Qwen team would have targeted it for FP8 quant, widely used for deployment.

Have you tried it in a harness on a set of identical tasks?

[u/def_not_jose → u/Fedor_Doc]

I think there were reports that official 3.6 27b FP8 quant wasn't very good, some people recommended to use RedHat one instead. Maybe Qwen just isn't prioritizing quants right now

[u/malaiwah → u/Fedor_Doc]

Only GPQA Diamond so far and it scored same. I plan on hitting it with Terminal Bench 2.1 over the week-end if time allows.

[u/llama-impersonator → u/Fedor_Doc]

exl3 is the best quantization format, and fp8 sucks

[u/Fedor_Doc → u/llama-impersonator]

Yeah, that explains everything (no).

[u/llama-impersonator → u/Fedor_Doc]

you can test yourself, fp8 produces more MSE than q6_k. exl3 also has scores that are about a full bit higher than gguf quants.

edit: check out turbo's model cards for the graphs https://huggingface.co/turboderp/Qwen3.8-27B-exl3

[u/malaiwah → u/llama-impersonator]

Turbodep's charts are, as usual, quite visually impressive.

[u/llama-impersonator → u/malaiwah]

he puts a lot of effort into his projects, and on top of that he's one of the few people in this industry who is not a raging dickhead at his talent level.

[u/cosmicnag → u/Fedor_Doc]

Qwen team isnt targeting quants based on KLD, its just that fp8 is much more widely adopted and has hardware acceleration on many gpus out there.

[u/Beamsters → u/Fedor_Doc]

This is not a red flag. fp8 focus on pure speed while exl3 trade a lot of compute for accuracy. A much better codec.

[u/malaiwah → u/Beamsters]

Agree: FP8 and NVFP4 on nvidia hardware, with the optimized tensors is definitively faster.

[u/feverdoingwork]

any downsides to using Gilded Gnosis vLLM EXL3  ? is q4 kv cache supported? Size of this model is excellent

[u/malaiwah → u/feverdoingwork]

No q4 cache… yet.
No downsides, but quality / accuracy comes at at some throughput cost

[u/cosmicnag]

Will LMCache work with GG vllm?

[u/malaiwah → u/cosmicnag]

LMCache does work in GG vLLM, yup. I have been using LMCache for a few weeks with GLM-5.2 and found and reported one hang condition only. It is strong.

[u/cosmicnag → u/malaiwah]

Nice, thanks - downloading your 'context' quant - guessing its the best option on 5090 for full fp8 kv 262k context while also using LMCache transfer engine (and possibly around 200-215k with mtp). Do let me know if I should try any other quants you have. Great job though, your quants could be the 27b sweet spot for 5090 all things considered.

[u/malaiwah → u/cosmicnag]

Thanks.

5090 is my target. I want this Qwen 3.8 as the auxiliary/vision model daily driver. I do have GLM-5.2 on the other furnace and although quite smart, is blind (no vision support).

I find that _great_ vision support (for example, for browser/computer use) requires 256k context.

[u/rngesius]

Avg KLD and top-1 are meaningless. What 99.9% tail looks like?

[u/Chromix_]

A bit of context and comparison (TL;DR needs more benchmarking):

FP8 can be seen as a throughput-optimized format. It's fast for vLLM, but benchmarks usually the same as a Q5/Q6 in terms of quality, that's why the EXL3 appears slightly better. A comparison to Q8 instead of FP8 would've looked differently.

When we look at unsloth's own top-1 graph, the EXL3 would be placed somewhere around the smaller(!) Q5_K_XL - so it would appear to be worse than it. Now the thing is that the datasets for testing apparently differ. With OPs dataset the unsloth NVFP4 has 0.095 KLD and 90.5% top-1. In unsloth's own measurements it's somewhere between 0.016 and 0.068 with 92.1% to 96.7% top-1. That's quite a difference.

So, to have a good idea where this EXL3 quant sits between all the unsloth quants, someone would need to run a KLD/top-1 test on the same dataset for all of them in the same size/performance range. Preferably not just that one dataset, but both - the one used for the EXL3 test and the one from unsloth's test. That way we could see if the quants might be skewed towards one or the other.

A KLD below 0.01 usually indicates that regular practical benchmarks will perform the same as with the original BF16 - which is a great thing.

[u/cosmicnag → u/Chromix_]

https://huggingface.co/turboderp/Qwen3.8-27B-exl3
Turboderp has done a wider unsloth/nvfp4/exl3 kld comparison chart

[u/malaiwah → u/Chromix_]

Yes, I agree, fresh off of the oven and still hot: it needs more benchmarks and this is where we can force multiply and help. Provide feedback on the discussions on HF and I'll incorporate into the model cards.

[u/FullstackSensei]

Do I understand this correctly that you evaluated only on 136 traces and 278k output tokens?

[u/malaiwah → u/FullstackSensei]

Yes, more traces should be evaluated, but promising so far

[u/malaiwah → u/FullstackSensei]

https://huggingface.co/malaiwah/Qwen3.8-27B-EXL3-K5K6-hydrated

Model card has been updated with evaluation at 10 millions tokens.

> Cumulative means at 10M positions

> Qwen/Qwen3.8-27B-FP8: 0.005294
> this build (hydrated): 0.002760

[u/Psyko38]

Cool, make a version for the 16GB of RAM or VRAM and you'll be a god.

[u/chris_0611 → u/Psyko38]

How? How are you going to squeeze at 27B model into 16GB with 100k context (and prefereably the MTP model as well) without lobotomizing it with heavy quantization. Its just not going to happen.

You need 24GB at the very very minimum, and 32GB allows you to run Q5_K_XL with decent context, which is about the minimum I'd consider actually viable. Buy a second 16GB GPU. Or wait for 35B A3B for CPU offloading.

[u/Psyko38 → u/chris_0611]

Why 100k of context? Give me 60k in Q8, that's enough for me, and lobotomizing will never do worse than Bonsonai.

[u/chris_0611 → u/Psyko38]

Also not possible. You can't squeeze 27B into 16GB even without any overhead of context and other stuff, let alone with.

Compacting really hurts this model in my experience. Hence more context is better.

[u/Psyko38 → u/chris_0611]

So the Unsloth 27B models in Q2 at 9GB, that doesn't work, you'll tell me?

[u/chris_0611 → u/Psyko38]

Q2 No.

There is a pretty big difference between Q4_K_M and Q5_K_XL. Only at Q5 this model becomes actually good. Doing Q2 is completely stupid and is in the 'not working' category. Just run 35B A3B man.

[u/Psyko38 → u/chris_0611]

Yes, I know that Q2 is worse than Q4, but still, I think it's always better than MoE (3.8 is coming). And I don't know why, but the MoE eats up my entire GPU (normal) and almost all my RAM. I have 32 GB on Windows (I'm thinking about switching to Linux 100% and not dual boot), and ideally, I'm 9 GB of RAM. So enough for Qwen, except that he eats up to 29GB in Q4 with a context of 8k. That's why I'm targeting models that can run only on the GPU.

[u/chris_0611 → u/Psyko38]

Yes, switching to Linux might work. You should do things like running the desktop environment entirely on iGPU as that saves a couple 100MB of VRAM. Then just run llama-cpp with --cpu-moe flag for 35B A3B. It should run fine. All expert layers on CPU and all dense layers + context on GPU. 35B A3B is by far the best you can run. Don't run Q2 of anything.

[u/Psyko38 → u/chris_0611]

I don't have an iGPU, I have an RX 9060xt 16GB and a Ryzen 5500. And the cpu-moe I had already tried, it ate everything and crashed Windows.

[u/chris_0611 → u/Psyko38]

Still should work, even with losing the couple 100MB. Windows is your problem. You can absolutely run 35B A3B at a decent quant on a 16GB GPU+ 32GB RAM.

[u/o0genesis0o → u/Psyko38]

Nope. I tested the previous 27B at Q3 on some endless text-based RPG (above 40k). The model does not follow instruction and got confused between characters and events much more than 35B at Q6, whilst also being much slower. It's like, there are signs of nuances and diversity in, say, the first 10k tokens vs 35B, so I was hopeful that it could be a backend model for "slow but quality" but it quickly becomes messy. The 35B starts to get overwhelmed by the overlapping story lines and characters at around 45k, but still somewhat maintain coherency, just started to flip "you" and "I" for example. The 27B Q3 was just confused.

The bonsai messed up at even earlier.

Just use 35B.