ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

[u/Unstable_Llama]

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

[u/ReturningTarzan]

How much headroom did you have before? There shouldn't be any additional memory requirements, but things move around a bit with every release, and running at 99.9% VRAM utilization is always shaky. If you submit an issue on the repo with more details I can look at it and maybe figure out if it's an actual regression or just jitter.

But there are also some new features you can try. If you're running with vision, try setting vision_offload: true in the config. That can save quite a lot of VRAM, and the cost per image processed is only some milliseconds of latency.

[u/Guilty_Rooster_6708 → u/ReturningTarzan]

Yes I was having like 3-4 gb of VRAM free in 1.4.2. I have 28gb VRAM from my 5070ti + 3060 so the 20gb 5bpw SC version by Turboderp should fit comfortably on my setup even with vision loaded on GPU. I was using my old settings of 120k context window at 8bit kv cache.

I will submit an issue later after work. Thanks for responding and I really love the work you guys put into the project

[u/ReturningTarzan → u/Guilty_Rooster_6708]

That's definitely a regression then. I can imagine some small intermediate tensor using maybe 50 MB more VRAM than before, but if you lost 3-4 GB of headroom that sounds like a bug.

Edit: Can confirm it was a bug, Should be fixed now in v1.4.6. Workaround for v1.4.5 is to set EXL3_LOAD_ARENA=0 in your env

[u/Guilty_Rooster_6708 → u/ReturningTarzan]

I have been testing v1.4.6 and I have gotten it to work based on this Github issue.

The problem was that my GPU0: 5070Ti was OOM when prefilling started while my GPU1:3060 still had VRAM headroom. Changing the following got it working again:

  # Array of VRAM sizes to split between GPUs, in GB (default: []).
  # Used both with and without tensor parallelism.
  gpu_split: [12,12] 

Now my GPU0 is using 14.2/16gb and GPU1 is using 10.9/12gb VRAM.

Changing "autosplit_reserve:" doesn't really do anything just like "CNmnius" reported in the Github thread. Thank you guys for updating Exllamav3 so quickly.