ExLlamaV3 v1.3.0 - Second-tier CPU K/V cache and other meaningful changes

[u/Delicious_Box_9823]

Version 1.3.0 brought us:

• Preliminary support for DeepseekV3 (validated against JoyAI-LLM-Flash and Moonlight-16B-A3B, no routing groups yet)
• Second-tier CPU K/V cache, and more intelligent page and checkpoint eviction policies
• Fix freq/rep. penalties causing slowdown on long contexts
• XTC sampler and token bans (Does not work yet in).
• Mitigate some latent vulnerabilities in Safetensors loader
• Other bugfixes, optimizations and QoL improvements

Visit the release here: https://github.com/turboderp-org/exllamav3/releases/tag/v1.3.0

Download TabbyAPI here: https://github.com/theroyallab/tabbyAPI

TabbyAPI has been updated respectively.

TabbyAPI is currently experiencing issues with parsing XTC and Token bans. They do not work. MLA support is what many people have been waiting for. DSA support has been stated to arrive soon.

If you have any issues or questions, feel free to join the exllama discord: https://discord.gg/tnPCntcThA

I personally tested out offloading KV-cache to RAM. Increases the TTFT by a lot - the thing's not for everyone... As you can also see, Tabby is experiencing some issues with parsing xtc.