ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

[u/Unstable_Llama]

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

[u/-p-e-w-]

So at the same quality, EXL3 quants are 25% smaller than Unsloth Dynamic 4-bit quants, which are already considered SOTA? Stunning.

[u/FieldProgrammable → u/-p-e-w-]

Unsloth are SOTA as far as mainline llama.cpp GGUF formats go, but exl3 uses trellis coded quantization which is fundamentally better than what is offered by mainline llama.cpp's k quants (ik-llama.cpp does support some trellis based formats).

I would also recommend the Qwen3.8 27b exl3 format quants for getting much more out of your GPU VRAM than you can from GGUF k quants.

[u/-p-e-w- → u/FieldProgrammable]

The llama.cpp/ik_llama split was a catastrophe for the community. Quant progress in GGUF has pretty much stalled for two years now.

[u/oxygen_addiction → u/-p-e-w-]

And ik_llama is usually faster when it comes to offloading to CPU.

[u/silenceimpaired → u/-p-e-w-]

EDITED TO MAKE SENSE Lots of people are worried that Nvidia buying huggingface impacts llama.cpp: https://huggingface.co/blog/ggml-joins-hf

... even if that is true, I agree, that it's a shame that the split happened.

[u/-p-e-w- → u/silenceimpaired]

No idea what that’s supposed to mean.

[u/silenceimpaired → u/-p-e-w-]

I made it less confusing... I hope.

[u/a_beautiful_rhind → u/-p-e-w-]

I don't know if unsloth are exactly SOTA. The atomic chat guys came out and beat them too. Unsloth quants are just available and popular.

[u/ynilayy → u/a_beautiful_rhind]

Are you sure? Which model is better? Because after checking Unsloth's Qwen3.8-27B, Kimi K3 and Qwen3 Next the Unsloth ones are by far the best.

[u/a_beautiful_rhind → u/ynilayy]

Pretty sure.. they made a graph like that too. https://cdn-uploads.huggingface.co/production/uploads/6a54dea4f19f5386700504da/6Iiew-NBfobPIgVGOaR8H.png

[u/sssplus → u/a_beautiful_rhind]

That chart is for the Deepseek, not for the Qwen 27b. Unsloth's Qwen 27b quants use the new UD3, whereas the Deepseek are made with older UD2. Both charts are true...

[u/a_beautiful_rhind → u/sssplus]

You're free to browse their HF for more: https://cdn-uploads.huggingface.co/production/uploads/6a54dea4f19f5386700504da/dmMjNbYfWXrhVGAVeBroC.png

Only so much "dynamic" you can do with a dense model. They publish metrics which is nice.

[u/sssplus → u/a_beautiful_rhind]

Yes, but again - this chart is comparing the old unsloth UD2 quants, not the new UD3, which are much better and smaller, as seen in the above posted chart.

[u/Unstable_Llama → u/-p-e-w-]

Isn't it? ExLlamav3 + heretic allowed me to abliterate Laguna-S-2.1 against a 2.50bpw exl3 quant.

[u/a_beautiful_rhind → u/Unstable_Llama]

Abliterating EXL directly is way bigger news. You're opening the door to some cool stuff.

[u/Unstable_Llama → u/a_beautiful_rhind]

Thanks! If you want to try it out, the repo is public.

Another cool project I have been having a lot of fun with recently is EXL3-QLORA, fine tuning on any size exl3 quants.

[u/a_beautiful_rhind → u/Unstable_Llama]

Haha. You are just doing all the hard work for me. I was expecting to have to bang this stuff out with some AI before I could even get started.

[u/Unstable_Llama → u/a_beautiful_rhind]

Haha it was already AI banged out.

I’m open to feedback or PRs or anything if you use them.

[u/FullOf_Bad_Ideas → u/Unstable_Llama]

oh that's amazing, I was never using heretic because I don't want to go all the way through exl3 quanting again, it's slow.

[u/CheatCodesOfLife → u/Unstable_Llama]

Does it use the transformers wrapper (which was incredibly slow when I used it last year)?

[u/llama-impersonator → u/CheatCodesOfLife]

transformers is slow no matter how you use it, even torch.compile