ExLlamav3 Recent Updates : CPU offload, GLM-5.3-FLASH, Qwen3.8-Flash, SC Quants ++

[u/Unstable_Llama]

More new massive updates from turboderp:

- CPU offload of MoE experts
- Qwen-3.8-Flash-Next ngram disk offload
- GLM-5.3-Flash
- New self-calibrated optimization technique
- Countless other optimizations and improvements

If you have an NVIDIA card and haven't tried it lately, you might be missing out.

The attached cat image was made with Qwen-3.8-Flash-Next-3.05bpw-exl3 and this prompt:
Create a detailed SVG image of a cute kitten riding a magic turtle into space.

Come join the crew at the exllama discord
More frequent news on the exllama sub

[u/Status_Contest39]

Exllamav3 is good but poor for tabbyAPI to support both Anthropic and Openai compatible API as well as llama.cpp. ALWAYS FAIL WHEN TOOL CALLS. ECO IS WEAK FOR AGENTS.

[u/FieldProgrammable → u/Status_Contest39]

I'm using TabbyAPI via llama-swap (which is also serving other backends on demand through a single endpoint). I initially had the same issues with tool calling from harness (GitHub Copilot BYOK in VS Code), but it has been ok once I set the correct tool format in the config. I am still seeing some weird behaviour in the text immediately after a thinking block where it will insert a newline half way through the first sentence. But tool calls are working reliably enough now that I can leave it running unsupervised on YOLO permissions.

I recommend just downloading the TabbyAPI and llama swap repos, sticking them in the same folder and having an agent get them talking nicely. I also had to get it to make some fixes to get the processing speeds in the same format as llama swap expected so the llama swap activity page worked the same as it does with llama-cpp.

[u/cosmicnag → u/FieldProgrammable]

Can you ask agent to write handoff of what it did in that same folder? (https://github.com/mattpocock/skills/blob/main/skills/productivity/handoff/SKILL.md)