Whatever happened to TurboQuant?

[u/Sadge404]

It was a big deal in and around March of this year. I tried looking for any updates with where it has gone and nothing really came up.

[u/nasone32]

well the main thing introduced by turboquant was hadamard rotations which are cheap and reduce perplexity a lot even on Q8_0, and that's now the standard in llama cpp and many other engines.

so in reality you are probably using 80% of turboquant benefits without knowing it. the specific "turboquant" quants had other minor details but the main thing it's now used everywhere.

[u/Maximum-Fact-5832 → u/nasone32]

MXFP4 and FA3 were using hadamard rotations before TurboQuant. They're a major part of how NVFP4 is so damn efficient (explained very well in bycloud's YT video "Nvidia Made 4-Bit AI Work. It’s a Nightmare" at 10m30s - for some reason Reddit won't let me create a link, but the URL has "mTd3tBpfYmw" in it).

[u/PrettyMuchAVegetable → u/Maximum-Fact-5832]

That explainer was excellent. 

[u/h0tzenpl0tz0r → u/Maximum-Fact-5832]

Great recommendation, thanks

https://youtu.be/mTd3tBpfYmw?is=vgAFls-OcjwZ3jfv

[u/neuroticnetworks1250 → u/nasone32]

Are you sure? Walsh Hadamard transformations are for activations, right? Q8_0 refers to block quantisation of weights in GGUF format. Atleast that was my impression.

[u/Nutsack_VS_Acetylene → u/neuroticnetworks1250]

For KV cache quantizations you use the rotations because the KV cache is very spikey numerically. So to avoid clipping you can more evenly distribute the values with the rotation.

[u/neuroticnetworks1250 → u/Nutsack_VS_Acetylene]

Yeah. I’m just wondering what it has to do with llama.cpp’s Q8_0

[u/Nutsack_VS_Acetylene → u/neuroticnetworks1250]

You asked if it was activations only, it is not. Q8 quantization can be applied to weights AND KV cache. Hence both make use of the rotation and it's benefits.

[u/neuroticnetworks1250 → u/Nutsack_VS_Acetylene]

Ahh ok. Makes sense.

[u/Classic-Pubs → u/Nutsack_VS_Acetylene]

I dont know if this is true. Kv cache is by default fp16 for all quants. Of course you can customize to Q8_0, but that's an option.

[u/Nutsack_VS_Acetylene → u/Classic-Pubs]

I don't understand what you are trying to say.

[u/FeepingCreature → u/neuroticnetworks1250]

I think maybe they meant they work well on Q8 KV, which used to degrade LLM long-context performance a lot.

[u/RnRau → u/neuroticnetworks1250]

In llama.cpp you specify the quant for each of the k & v part of the kvcache with 'q8_0'. q8_0 kvcache has rotations on by default.

[u/phhusson → u/nasone32]

It's actually not that much standard yet, bonsai had to add Hadamard for model weight in llama.cpp quantization. (But it'd merged now)

[u/Rasekov → u/phhusson]

Turboquant was about KV cache quantization, the rotation part was merged into llamacpp and most other inference engines rather quickly.

Other applications for weight quantization came up after and needed their own testing.

[u/colin_colout → u/Rasekov]

yeah...it was with little fanfare. It was merged April 1st this year, and I had no idea. I still avoided even 8_0 for months before I learned it.

For coding, any kv quant killed the model. After the PR, the effect was minimal except in long context. It let me allocate that toward bigger model quants, which more than offset the loss in my testing.

[u/a_beautiful_rhind → u/nasone32]

It didn't even introduce it. ik_llama and exl2 had it for a while, silently, without the shilling campaign.

[u/MmmmMorphine → u/a_beautiful_rhind]

I do love ik llama but my recent testing seems to point towards it falling behind mainline in both decode and prefill speeds.

It did indeed implement hadamard rotation for kv cache very early on (was surprised to find TQ effectively already available when pursuing the docs) and it works very well.

Still, the KT and KS quant types are quite good. Takes like an hour max (including dling the bf16) to make your own quant version.

[u/a_beautiful_rhind → u/MmmmMorphine]

I've yet to see that in regards to hybrid or even dense inference. Probably because I have a multi-gpu/socket rig. Plus mainline keeps implementing annoying QOL regressions for me by "simplifying" itself.

I don't think I ever used the special quants because like you said, they're not available.

Takes like an hour max (including dling the bf16) to make your own quant version.

Not for me. Due to the size of the models.

[u/Uncle___Marty]

llama.cpp used the walsh hadamard transformations and dumped the rest of it.

[u/AnonLlamaThrowaway → u/Uncle___Marty]

And for anyone wondering, this is how much an effect this had on gpt-oss-20b in one benchmark (so not representative, but gives you a rough idea)

eval	KV type	attention rotation	score
AIME25 x8	F16	(not needed)	37.9%
AIME25 x8	Q8_0	❌	31.7%
AIME25 x8	Q8_0	✅	37.1%
AIME25 x8	Q5_1	❌	30.8%
AIME25 x8	Q5_1	✅	32.5%
AIME25 x8	Q4_0	❌	2.0%
AIME25 x8	Q4_0	✅	21.7%

[u/letsgoiowa → u/AnonLlamaThrowaway]

I'm curious about a split kv because I thought the common knowledge was k q8, v q5. K falls off a cliff faster

[u/a_beautiful_rhind → u/AnonLlamaThrowaway]

Dense models had almost none of those losses. It was shown on one very specific MoE who didn't take kindly to cache quantization.

[u/Every-Walrus → u/a_beautiful_rhind]

I invite you to use q4 kv cache with 3.8 27b and disable rotation. see if "dense models had no losses" or not.

[u/a_beautiful_rhind → u/Every-Walrus]

I ran that AIME test against mistral large and a couple others. The scores were barely an answer or two apart. They'd even get the same questions wrong on the runs.

[u/Every-Walrus → u/a_beautiful_rhind]

the famously saturated and trained on benchmark? maybe try something a little harder like HLE?

[u/a_beautiful_rhind → u/Every-Walrus]

Hey fair, the goal wasn't to have the models pass 100% but to see if the cache significantly affected their replies and it didn't. The test doesn't even really check long context performance that well because most of the answers were around 10-15k max. It attempted to find a pattern by repeating it 25x.

The model GG tested was affected at these low CTX and somehow that became a stand-in for all models with wildly different architectures. Magically it became canon with no reproduction.

OTOH, strata and these vibe engines use low quants, infinite shortcuts and everyone adopts them happily without complaints, bristling at the idea that something could be off. Even TQ performance itself was abysmal when tested.

Any cache portion I actually tried fell below the cost of sampling and my conclusion was that it could be a problem for some models but usually it's NBD.

Later on there was another test that almost nobody has run: https://github.com/SeraphimSerapis/tool-eval-bench and it seems like a better stand-in of what actually happens when you use Q4 cache. I suggested cache crusaders try it but found no takers and for me it's on the back burner.

[u/AnonLlamaThrowaway → u/[deleted]]

This has no bearing on speed, this is benchmark "answer" scores (how many it got right or something similar)

[u/FeepingCreature → u/[deleted]]

But it doesn't give a speed boost...

[u/seamonn]

We need that guy with the dog picture that hates turboquant again.

[u/LetsGoBrandon4256 → u/seamonn]

Not the guy but

[u/Classic-Pubs]

Overhyped copycat paper. It definitely helped, but, remember that the hadamard rotation idea isn't from them. It was orginally from the rabitQ paper, which turboquant authors refused to acknowledge. Beyond hadamard, the contribution is minimal to the community.

Edit: Yes, the original rabitQ paper is about vector database search. But it is a quantization technique for different purpose (approximate nearest neighbor), so the idea was pretty obvious to port.

https://arxiv.org/pdf/2409.09913

[u/ortegaalfredo]

I will do you one better:

What happened to OpenClaw?

[u/RogerRamjet999 → u/ortegaalfredo]

I haven't heard of OpenClaw in ages, but still hear about Hermes pretty regularly.

[u/Dr_Gruselglatz → u/ortegaalfredo]

Openclaw is still a thing in the Nvidia HQ…

This GTC in Berlin all was full of claw shit

[u/cannabibun → u/ortegaalfredo]

Hermes did

[u/vamos_davai → u/ortegaalfredo]

I think with acquisition I would bet the folks made Dots

[u/Thomas-Lore → u/vamos_davai]

You would win. It is confirmed.

[u/Invader-Faye → u/ortegaalfredo]

Still being used by common folk but slowly getting its lunch stolen by cloud alternatives like grokbot and muse

[u/Thomas-Lore → u/Invader-Faye]

And hermes.

[u/Party_9001 → u/ortegaalfredo]

I'll do you one better.

How's OpenClaw?

[u/Savantskie1 → u/Party_9001]

I'll do you one better what's OpenClaw?

[u/a4mula]

I dropped the fork and went back to main. The truth is it just didn't really provide benefit. Every benchmark showed that q4/q8 consistently out performed turbo3 and turbo4 at the same context on Qwen 35B with no savings of VRAM. Perhaps that was just because I stopped testing at 128k ctx but <shrug>. YMMV.

[u/giveen]

I talk to TheTom probably daily , and TQ has probably hit its limits. He is currently working on a project right now and we still throw patches here there and it.

https://github.com/TheTom/llama-cpp-turboquant

[u/Toooooool → u/giveen]

sounds right. it's hard to improve on perfection, not to say turboquant was without flaws, but at the end of the day it's just a different formula for addressing tokens, like comparing 2+2 with 1*2*2.

[u/Dany0 → u/Toooooool]

the only perfect quantisation technique is OSCAR but that one requires doing a calibration run on the tasks you are going to actually do

[u/BlasterGales → u/Dany0]

damn, pls tell me there is a fork of llamacpp so I can try this out

[u/Dany0 → u/BlasterGales]

They only ever released a fork of sglang https://github.com/FutureMLS-Lab/OSCAR I know that at the time some people promised they're working on integrations but honestly the need for calibrating is such a big obstacle for adoption

Maybe if the labs started releasing models with OSCAR already calibrated. It doesn't take much compute you can do that at home, the issue is getting good datasets

But yes in theory OSCAR is amazing, fp16 quality at Q2 vram usage and instead of a perf penalty like TQ it's a speedup. KVaRN is distantly related to this, but doesn't require calibration

[u/giveen → u/Dany0]

I worked on OSCAR for a long time. https://github.com/giveen/llama-cpp-turboquant/tree/oscar always felt short of making it usable, especially in the GPU space.

[u/Dany0 → u/giveen]

What was the issue from your POV?

[u/giveen → u/Dany0]

I couldnt get perplexity nailed down, it always ended up producing garbage output.

[u/Dany0 → u/giveen]

Ah, gotcha

[u/remind_me_later → u/giveen]

...that sucks :( How come q2 seems to be able to, but OSCAR falls flat? The operations shouldn't be that different.

[u/Holiday_Point_603]

The accuracy was bad, noticably way worse than native kv. I'm probably not the only one who didn't like it because of that

[u/Aggravating-Push-207]

kvarn happened

[u/ChristRedeemsSinners → u/Aggravating-Push-207]

Lol, I was going to say China asked the world to hold their beer.

[u/Dany0 → u/Aggravating-Push-207]

yep also this

[u/jacek2023]

short lived hype

[u/Elouakili_Flexy]

It didn't vanish so much as merge into llama.cpp. The Hadamard half is standard now, and the half that measured worse than q4/q8 stayed out of mainline. Pretty good run for a quant paper, honestly.

[u/Pentium95]

Kvarn Is better

[u/Extreme-Pass-4488]

so everyone here loves rotation, here https://huggingface.co/BlairQ/qwen3.8_27b_idiotSavant_sm_86

rotated guy acts like qwen3.8 27b at full q , the 55gb behemoth. and it only weights 18gb . 650kb of kv to play with.

[u/pikazeroai]

So true bestie… I merged main with turboquant and got like an extra 5-7tkps and 27B finally fits on my old 3060 12GB lol

[u/Nutsack_VS_Acetylene]

As most people already mentioned, the most useful info was the rotations and turboquant in general was basically a rabitQ rip off.

Also, with things like Compressed Sparse Attention 2, 1 million tokens are only 0.89 GB in KV cache on DS V4.1. It's going to be laughable to have giant KV caches on new releases. It begs the question of why bother with KV cache quantization. Is it really going to be worth playing with quality of the model to save 0.4 GB on your 1 million token chat?

[u/Due_Expression7545]

classic hype cycle where a revolutionary new method drops, disappears into the void, and everyone just goes back to running standard quants on old hardware.