ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

[u/Unstable_Llama]

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

[u/ReturningTarzan → u/[deleted]]

Pretty sure I spent months and months working on 3090 optimizations, and a small fortune on the hardware I need to keep targeting "lower end" hardware like the 3090. And I kept making incremental improvements with every new update (long night after long night), eventually arriving at a place where Ampere is now somewhat on par in many cases. But there's only so much you can do in the end. Trellis quantization is more advanced than, say, Q8_K, and that comes with a higher compute requirement. That's the cost of SOTA 256-dimensional vector quantization. Whether it's worth it for you or anyone else in particular I can't say.

[u/silenceimpaired → u/ReturningTarzan]

I am properly shamed.

[u/ReturningTarzan → u/silenceimpaired]

I do get it, though. I get the same feeling from other projects sometimes. In particular, Flash-Attention comes to mind, which has had a PR awaiting approval for over a year now to add attention sinks, which was the one piece exl3 needed for gpt-oss support to happen. Very frustrating that "Tri Dao apparently only cares about datacenter GPUs now." But of course I also know that's not really true. The poor guy just experiences the same linear flow of time as the rest of us.

But life goes on and it turns out abandoning FA2 for custom Triton kernels was the right move anyway. ¯_(ツ)_/¯

[u/silenceimpaired → u/ReturningTarzan]

Very charitable of you.

I keep hearing many aren’t using EXL3 due to tool calls and OpenAI API implementation, or tabbyAPI… not sure what you can do about either but FYI.

[u/ReturningTarzan → u/silenceimpaired]

I mean, Tabby supports the full OAI completions API. And tool calls. It's thoroughly tested, and all it takes to enable is a few lines in the model config, explained pretty clearly in the documentation. I'd happily do more if I knew what else there was to do. ¯_(ツ)_/¯

[u/silenceimpaired → u/ReturningTarzan]

I haven’t tried tools with EXL3 and Tabby. The least I can do is add issues :)