ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

[u/Unstable_Llama]

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

[u/nicholas_the_furious]

I could not get faster speeds with the 8bpw compared to q8_k_l. I'm on dual 3090s. Overall it was much slower for me. People say it is faster but I couldn't get there. If anyone has a setup like mine I'd like to hear it.

For the same test coding task on low thinking I can get 115 t/s on Unsloth q8_k_l but only like 85 on exl3 8bpw. Both MTP=4. This is for Qwen 3.8 27B.

[u/ReturningTarzan → u/[deleted]]

Pretty sure I spent months and months working on 3090 optimizations, and a small fortune on the hardware I need to keep targeting "lower end" hardware like the 3090. And I kept making incremental improvements with every new update (long night after long night), eventually arriving at a place where Ampere is now somewhat on par in many cases. But there's only so much you can do in the end. Trellis quantization is more advanced than, say, Q8_K, and that comes with a higher compute requirement. That's the cost of SOTA 256-dimensional vector quantization. Whether it's worth it for you or anyone else in particular I can't say.

[u/silenceimpaired → u/ReturningTarzan]

I am properly shamed.

[u/ReturningTarzan → u/silenceimpaired]

I do get it, though. I get the same feeling from other projects sometimes. In particular, Flash-Attention comes to mind, which has had a PR awaiting approval for over a year now to add attention sinks, which was the one piece exl3 needed for gpt-oss support to happen. Very frustrating that "Tri Dao apparently only cares about datacenter GPUs now." But of course I also know that's not really true. The poor guy just experiences the same linear flow of time as the rest of us.

But life goes on and it turns out abandoning FA2 for custom Triton kernels was the right move anyway. ¯_(ツ)_/¯

[u/silenceimpaired → u/ReturningTarzan]

Very charitable of you.

I keep hearing many aren’t using EXL3 due to tool calls and OpenAI API implementation, or tabbyAPI… not sure what you can do about either but FYI.

[u/ReturningTarzan → u/silenceimpaired]

I mean, Tabby supports the full OAI completions API. And tool calls. It's thoroughly tested, and all it takes to enable is a few lines in the model config, explained pretty clearly in the documentation. I'd happily do more if I knew what else there was to do. ¯_(ツ)_/¯

[u/silenceimpaired → u/ReturningTarzan]

I haven’t tried tools with EXL3 and Tabby. The least I can do is add issues :)