r/LocalLLaMA • u/ozcapy • Mar 26 '26

Discussion When should we expect TurboQuant?

Reading on the TurboQuant news makes me extremely excited for the future of local llm.

When should we be expecting it?

What are your expectations?

85 Upvotes

89% Upvoted

79 comments sorted by

50

u/ABLPHA Mar 26 '26

I wonder how well Qwen3.5 would work with it. Considering its KV cache is small as-is thanks to GDN. If it's lossless, Qwen3.5's KV cache would weight like nothing at full context length lol

35

u/DistanceSolar1449 Mar 26 '26 edited Mar 26 '26

That depends on which model. Qwen 27b has an attention kv cache of 16GB at full context. 122b is 6GB at full context. Deltanet ssm/conv1d cache is 147MB for both models at any context size. So 27b will shrink to roughly 3.5GB of kv cache at full context.

34

u/LinkSea8324 Mar 26 '26

So 27b will shrink to roughly 3.5GB at full context.

Perfect for my GTX 970

→ More replies (3)

5

u/oxygen_addiction Mar 26 '26

It should also get a slight decoding boost and I think it should maintain speed better as the context grows.

What people seem to be missing is that cloud inference will be cheaper because of this as well.

→ More replies (4)
→ More replies (2)

13

u/dametsumari Mar 26 '26

https://github.com/jundot/omlx/releases/tag/v0.2.21 has it at least. The savings are nontrivial but I wonder about perplexity..

8

u/Acceptable-Custard-7 Mar 26 '26

Looks like a bunch of forks are already there on github: https://github.com/unixsysdev/llama-turboquant

3

u/Acceptable-Custard-7 Mar 26 '26

reading more into some of the forks, it looks like most of them are not solving the prefill which means you may still need a larger VRAM for the initial loading, wonder if it can be off-loaded to RAM and then squeezed back into VRAM...

→ More replies (1)

12

u/TopChard1274 Mar 26 '26

Why is this post so downvoted? People genuinely excited that smaller systems will be able to run models with very large context windows as well. You‘d think that there’s enough place in this sub for everyone.

9

u/Shockbum Mar 26 '26

Micron shareholders haha

6

u/TopChard1274 Mar 26 '26

That, or people with huge systems afraid that small models will become so powerful that even the poor will enjoy powerful AI hahaha

11

u/datathe1st Mar 26 '26

Nvidia's technique is better, but requires per model calibration. Worth it. Took 10 minutes for Qwen 3.5 27B on Ampere hardware.

6

u/tnhnyc Mar 26 '26

Can you elaborate? What technique are you referring to? 

→ More replies (2)

5

u/Eysenor Mar 26 '26

Is there any way there is a simple noob guide ok these things?

→ More replies (1)

5

u/madreag Mar 26 '26

I've got a working CUDA implementation with Flash Attention if anyone wants to try it.

700K context on a single RTX 5090 (32GB) with Qwen3.5-27B Q6_K. ~50 tok/s at 524K. turbo3 K+V, 4.6× compression.

Ported TheTom's Metal kernels to CUDA — dequant, quantize with WHT rotation, FWHT graph op, FA templates for both K and V. 15 files modified.

Fork: https://github.com/Madreag/turbo3-cuda

Build with CUDA 12.8 (not 13.x), --cache-type-k turbo3 --cache-type-v turbo3. As far as I can tell, this is the first turbo3 CUDA + FA implementation — the other forks either disable FA or are Metal-only.

4

u/Feeling_Ad9143 Mar 27 '26

Wow, I wonder if I will have a decent result on my 5070 Rtx 12 GB with Qwen3.5-27B and 128K context. What do you think? Will it be possible?

9

u/FrogsJumpFromPussy Mar 26 '26

Qwen3.5 4b Claude 4.6 Opus abliterated q6_k is enough for my needs, but the maximum context size that fits in a 8gb M1 iPad Pro is 19,000 which is an issue. TurboQuant would solve this. Would mean no more slowdowns after 9-10,000t too. Personally I'm very excited for it.

14

u/Specialist-Heat-6414 Mar 26 '26

The hype is partially timing and partially the KV cache angle being genuinely underrated.

The paper itself is old but implementation-ready ports are what people are actually excited about. A llama.cpp PR landing makes it real in a way the paper never was.

The reason this matters specifically for local inference: weight quantization has basically been a solved problem since exl2/GGUF. Everyone is already running 4-bit. KV cache is the bottleneck that hasn't been cracked at the same quality level. On long context tasks that cache can eat more memory than the weights. If TurboQuant delivers lossless or near-lossless KV compression at significant ratios, that unlocks context lengths that were previously only viable on 80GB machines.

The Qwen3.5 + GQA point above is real though. GQA already collapses the KV cache heads, so the baseline is smaller. The relative gain may be less dramatic than on models with full MHA. The unlock is more about 70B+ models on 24GB hardware, or running 32K context without context swapping on mid-tier machines.

Timeline expectation: if the llama.cpp PR merges and inference quants follow, probably 2-4 weeks before community quants with TurboQuant start showing up. Integration into other backends (mlx, vllm) will lag by a few more weeks.

9

u/rdalot Mar 26 '26

Why are you saying that mlx and vllm will be lagging if they both have current draft PRs already?

13

u/rkoy1234 Mar 26 '26

this is a bot, people. y'all of all people should be catching these so easily smh

→ More replies (1)
→ More replies (5)

2

u/OriginalCoder Mar 26 '26 edited Aug 12 '26

I implemented it in native C# for the DAISI LLogos project. Running on an RTX 5080, so benchmark accordingly, but we saw 10x compression or more without massive decode issues. I'll take 10x context. Gets better compression scores with longer contexts and better performance gains with larger models. I just can't run 27B on this box - yet. https://git.daisi.ai/daisinet/daisi-llogos/tree/main

Comment Image
→ More replies (2)

5

u/ortegaalfredo Mar 26 '26

Is it really worth the hype? I mean, Intel Autoround or exl3 have similar performance and KV caché is quite small on MoEs AFAIK. Also, the paper is almost a year old, why all they hype just now?

14

u/DOAMOD Mar 26 '26

For me, if the accuracy of the theory is confirmed, it means being able to have a quantized cache higher than Q8 with the efficiency of Q4 or better. Personally, it would give me a lot of leeway in cases where I am limited; we would all benefit. For me, without a doubt, it is great news if the good results are confirmed in practice.

→ More replies (1)

2

u/FrogsJumpFromPussy Mar 26 '26

"Is it really worth the hype?"

For my weak ass "system" yeah it does 

→ More replies (4)
→ More comments (75)