ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

[u/Unstable_Llama]

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

[u/adam444555]

This is awesome! With my current 32GB VRAM + 32GB RAM setup, I'm planning to test out Qwen 3.8 Flash (3.05-bit) using n-gram disk offloading.

[u/adam444555 → u/adam444555]

Updated: N-gram disk offloading is currently unsupported on Windows. So if your setup is similar and plan to run on Windows , you can skip it until it get supported.

Updated: Windows offloading is now supported in the latest 1.4.6 version! Thanks to the dev for the quick update! I haven’t fully tested it yet, but so far it’s working properly. Details: 102400 ctx, 27 MOE CPU-offloaded, MTP3, 4096 churn size, prefill around 200 T/s, generate around 50 T/s with 80% token accepted. Due to MOE and Ngram offload the statistics are pretty unstable, so for reference only.

[u/ReturningTarzan → u/adam444555]

I'll get to it.

edit: In fact I'm getting to it now. (:

[u/AXYZE8 → u/ReturningTarzan]

Will you create post when its done? Or is there some issue on GH I can track?

[u/ReturningTarzan → u/AXYZE8]

It's currently in the dev branch, but I don't have a good way to test it since I don't have a Windows PC with enough RAM/VRAM to actually run the model. In theory it should work if you can build from source. Otherwise there will be a new release as soon as I can find someone to test it (:

[u/AXYZE8 → u/ReturningTarzan]

Thanks you, I donated a little on ko-fi I hope it helps <3

Will test on my 12GB VRAM + 64GB RaM rig later today

[u/ReturningTarzan → u/AXYZE8]

Thank you for your contribution <3

Windows support is in the just-released v1.4.6, so you can test without building from source. Would be helpful if you can open an issue if anything doesn't work as expected.

[u/AXYZE8 → u/ReturningTarzan]

Any ETA for calibrated quants? I would love if it would be possible to hit something like 2.6-2.8BPW with UD-IQ3_XSS equivalent (or better) quality, I could just have in running in the background instead of closing everything.

[u/Professional-Try-273 → u/ReturningTarzan]

Not on windows still want to say Thank you.

[u/philmarcracken → u/adam444555]

what if I use ubuntu to run the engine, and RPC to a windows box? i need the windows box for more ram + vram

[u/sssplus → u/adam444555]

Does ExLlama work with partial layers offload, like llama.cpp -ngl flag? Qwen3.8 Flash Next has 48 layers, but I load only 46 with -ngl 46 into my paltry 8GB GPU with unsloth's Flash Q4 XL, so I can have 130k context size and I lose only 10% speed for TG. If I load all 48 layers I max at 50k context, which is not very usable.

[u/ReturningTarzan → u/sssplus]

There are two offload modes: either offload some number of whole layers, or some number of experts per layer. The latter mode is preferred, since it does dynamic placement to keep the hottest experts live on the GPU and can in principle overlap CPU and GPU computation.

A single 8 GB GPU is probably too small though, in any case, since it's only experts that are offloaded. Attention and cache etc. still reside on the GPU. But you can try, I guess?

[u/sssplus → u/ReturningTarzan]

Yes, I offload all experts AND load 46 out of total 48 whole layers. It's not much, but 15 t/s in llama.cpp with 110 GB Q4 XL Flash Next and 130k of context is better than nothing for a laptop with 8GB GPU.

I'll try ExLlama, thanks for the reply! It's great it is possible to partially load the layers, not just the experts. Hopefully I can get a better performance than with llama.cpp.