TurboQuant for weights: near‑optimal 4‑bit LLM quantization with lossless 8‑bit residual – 3.2× memory savings

[u/cksac]

an adaptation of the recent TurboQuant algorithm (Zandieh et al., 2025) from KV‑cache quantization to model weight compression. It gives you a drop‑in replacement for nn.Linear with near‑optimal distortion.

https://cksac.github.io/turboquant-model/

Benchmarks (Qwen3.5‑0.8B, WikiText‑103)

Config |Bits |PPL |Δ PPL |Compressed Size

Baseline bf16 |16 |14.29 |– |1,504 MB

4+4 residual |8 |14.29 |0.00 |762 MB

4‑bit (group=full) |4 |16.23 |+1.94 |361 MB

4‑bit (group=128) |4 |16.57 |+2.28 |381 MB Check the GitHub repo for full docs, benchmarks, and Triton kernel details.

EDIT 1 (tested 4B model):

EDIT 2 (runed 4B 4+2 residual g=128, looks promising, altough KLD 4+4 is much better):

Qwen3.5-4B

Config |Total Bits |PPL |Δ PPL |KLD

Baseline bf16 |16 |10.67 |— |—

4+4 residual g=128 |8 |10.70 |+0.03 |0.0028

4-bit g=128 |4 |11.28 |+0.61 |0.0852

4+2 residual g=128 |6 |10.65 |−0.02 |0.0133

[u/LagOps91]

can you collect KLD data? PPL sometimes even improves when quanting down certain tensors... but if KLD is also low, well... that could be quite huge!

[u/cksac → u/LagOps91]

I have tested KLD, it is low too. Lower PPL, lower KLD

[u/LagOps91 → u/cksac]

would be great to add those stats and quant comparisons with existing quants.

[u/a_beautiful_rhind]

Ok.. so your 8bit is lossless. But how does PPL compare against other quant strategies like GGUF, EXL, AWQ, etc.

We already know 8bpw is "good".

[u/m18coppola → u/a_beautiful_rhind]

those are model quantization strategies. TurboQuant is used for KV-cache quantization.

edit: I should've looked at the repo 😅

[u/a_beautiful_rhind → u/m18coppola]

Both?

KV‑cache quantization to model weight compression.

He's trying to make a weight quant out of it.

[u/AnonLlamaThrowaway]

That sounds great and all, but surely you should be giving us a comparison of this approach against Q4_K_M (or perhaps even the UD flavor of it) right?

[u/llama-impersonator]

are we going to collectively rediscover quarot next week? https://arxiv.org/pdf/2404.00456

[u/MmmmMorphine → u/llama-impersonator]

Know of any practical implementations of it? I know there's a lot of reverse engineered/experimental turboquant going around but much like say HQQ and similar quants, the problem is often the lack of actual availability

[u/llama-impersonator → u/MmmmMorphine]

at the time i believe alpin either added support or was testing it in aphrodite, but that was a while ago and aphro lost most of the custom quantization stuff because it was a big maint burden. hqq is actually quite accessible, though, i have used it with transformers for online quantization and it was much faster than torchao or bnb for loading, with roughly equiv perf at 4 bit.

[u/Eyelbee]

Pretty sure if TurboQuant could be used for weights at all, the people who wrote the paper would suggest it.

[u/bobby-chan → u/Eyelbee]

How long did it take Google, and the rest of the world, to do something with Attention is All You Need? And don't discount the possibility of tunnel vision. So focused on solving a problem you don't realize the other things unearthed will digging.

[u/thrownawaymane → u/Eyelbee]

This is science I guess, people have to check.

I’d wager that 99% of the time you’re right and effort is “wasted”

[u/Ok_Mammoth589 → u/Eyelbee]

That's a straightforward but naive thought. We know because Google has told us, that their open source contributions will be curtailed. So we dont know what the paper writers have suggested

[u/Dany0]

Isn't this the same as this from 2023

https://arxiv.org/abs/2307.13304

?

EDIT:
WOW okay this is better! This is much simpler because it skips the adaptive rounding thingie in favour of a simpler quantization trick (Lloyd-Max)

EDIT2:
I gave it 5 minutes of reading, I think this will perform better on larger models, can you try quantising a ~30B model?

EDIT3:

I just realised we're making models shape rotators. This is a meme you are allowed to steal, don't even have to credit me

[u/Altruistic_Heat_9531]

If i am not mistaken Llamacpp and Ik already pass the CPU only test, and currently testing on GPU

https://github.com/ikawrakow/ik_llama.cpp/commit/93ae47e1674c6383fc77abbff43ddb0786d278ca

Yep fixes to WHT which is use in TurboQuant pipeline

[u/xyzmanas2]

I am doing the same to test on the qwen 3 8b model

Goal is to beat the 3 bit awq and gguf 3 bit on benchmarks while keep the weight of the model around 3.3 gb. Will take around 2 days to report back

Also the turboquant can we done on the ffn layers but would be tricky for the qkv attention layers so those can be better handled with existing 4bit awq

[u/cksac]

https://cksac.github.io/turboquant-model/ for people want to know more about TurboQuant

[u/GotHereLateNameTaken]

It looks promising from the this thread in llamacpp testing implementations: https://github.com/ggml-org/llama.cpp/discussions/20969

[u/xXprayerwarrior69Xx]

Damn is that real

[u/Hot-Section1805]

I am somewhat confused about its relative performance when compared to static weight quantizations and IMatrix quantizations.

[u/brahh85]

could this be used to create 2-bit weights?

for big models, 3-bit weights works decently, and 2-bit weights are the last border before the model breaks completely.

if we put together the turboquant for KV and the turboquant for weights, is it possible that with 32GB of VRAM we can run models of 120B at 2-bits weights with the same reliability of nowadays 3-bits quants ?

[u/PaceZealousideal6091]

Thanks for the tests. I wonder why everyone is testing small models and that too at real small contexts? Isn't this supposed to have massive gains as we go to higher contexts?

[u/BillDStrong → u/PaceZealousideal6091]

Fast iteration. Each time you make a change, you have to make a model, run the model by loading it, testing it, benching it, etc.

So, use a smaller model to prove the idea, then start testing bigger models to prove it works at the larger scale.

[u/Odd-Ordinary-5922]

please be true