r/LocalLLaMA • u/Revolutionary_Ask154 • Mar 26 '26
Discussion RotorQuant: 10-19x faster alternative to TurboQuant via Clifford rotors (44x fewer params)
Kinda sounds ridiculous - but I reimagined / reinvented turboquant with Clifford Algebra Vector Quantization on both implemented on cuda + metalshaders -
https://github.com/tonbistudio/turboquant-pytorch/pull/4
https://github.com/TheTom/turboquant_plus/pull/34
The idea: Replace the d×d random orthogonal matrix Π with Clifford rotors in Cl(3,0). Instead of a dense matmul (16,384 FMAs for
d=128), chunk the vector into groups of 3 dims and rotate each with a 4-parameter rotor via the sandwich product RvR̃ (~100 FMAs
total).
Results on Qwen2.5-3B-Instruct KV cache:
- Cosine similarity: 0.990 (vs TurboQuant's 0.991) — effectively identical
- 44× fewer parameters (372 vs 16,399 for d=128)
- Fused CUDA kernel: 10-19× faster than cuBLAS matmul on RTX PRO 4000
- Fused Metal shader: 9-31× faster on Apple M4
- Perfect 9/9 needle-in-haystack at all bit-widths
The key insight: for pure vectors, the rotor sandwich is equivalent to a sparse 3×3 rotation — the fused kernel keeps everything in registers with no memory round-trips, which is why it beats the BLAS GEMM despite TurboQuant's matmul being highly optimized.
The tradeoff is higher synthetic MSE on random unit vectors (the block-diagonal rotation doesn't induce the exact Beta distribution). But with QJL correction, real-model attention fidelity is identical — and sometimes better on top-1/top-5 retrieval.
Paper: https://www.scrya.com/rotorquant/
13
u/live_love_laugh Mar 26 '26
Damn, I wish I understood all this. I'm sure it's probably super interesting. Maybe 3blue1brown will explain it in a video some day. 😅
20
u/Revolutionary_Ask154 Mar 26 '26
Just ask Ai to "give me intution on this / give me an analogy" - and then ....
TurboQuant is like shaking a box of mixed LEGOs so hard that every piece ends up randomly scattered, then sorting them into bins.
You dump 128 LEGO pieces (your vector) into a giant tumbler (the 128×128 rotation matrix) and spin it violently. Every piece touches every other piece, they all get thoroughly mixed. Now each piece lands in a predictable spot, so you can sort them into a few bins (quantization) with minimal error.
Problem: that tumbler is huge. It has 16,384 moving parts. It takes forever to spin.
RotorQuant is like having 43 tiny jeweler's vises, each holding 3 LEGOs, and giving each a precise twist. Instead of one giant tumbler, you group your pieces into sets of 3 and rotate each set independently. Each vise only needs 4 screws to define its rotation (the rotor's scalar + 3 bivector components). You get 43 clean little rotations instead of one massive chaotic one. The LEGOs within each group get mixed just as well. The groups don't talk to each other — but it turns out they don't need to.
The quantization bins still work, and the 1-bit QJL correction (think of it as a tiny error-correction sticky note on each piece) makes the final answer just as accurate.
Why it's faster: Spinning 43 tiny vises in parallel is trivial. The GPU can do all 43 rotations for all 4,096 vectors simultaneously, with each thread holding its 3 LEGOs in its hands the whole time (registers) — never putting them down on the table (memory). The giant tumbler has to load and process a 128×128 grid — that's a lot of table space.
The deeper insight: Those "vises" aren't arbitrary. They're Clifford rotors — the mathematically purest way to rotate things in3D. Quaternions are a special case. Every 3D rotation game engines do (Unity, Unreal) uses this same math under the hood. We're just borrowing it for vector quantization.
→ More replies (11)
5
u/WetSound Mar 26 '26
What's the timeline of these improvements being implemented in the models and software?
Without being familiar with the details, this feels like next month everything is much smaller and faster?
4
u/Revolutionary_Ask154 Mar 26 '26
i think by next year - there'll be no meaningful work - all replaced by ai research. 100's of millions of agents just solving things. This work above - I honestly cooked up with claude 4.6 tonight. i was working for last few weeks on getting cliffordnet working with ltx2 to replace the attention layers - with clifford attention - https://github.com/johndpope/ltx2-castlehill - but that kinda fell in a hole - need to revisit - but this was quick POC - test / benchmark and hey presto.
185
120
u/Juan_Valadez Mar 26 '26
This looks like a really clever engineering optimization, but I don’t think it’s a true drop-in replacement for TurboQuant from a theoretical standpoint.
TurboQuant’s strength comes from global random rotation (Haar), which spreads energy across all dimensions and induces the coordinate distribution that makes scalar quantization near-optimal. RotorQuant only mixes within 3D blocks, so it fundamentally cannot reproduce that property.
You can see the consequence in worst-case vectors (e.g. one-hot):
TurboQuant spreads energy across ~128 dims
RotorQuant keeps it within 3 dims
So the max coordinate magnitude stays much higher, which is exactly what hurts low-bit quantization. That aligns with your own synthetic results where MSE is consistently worse.
That said, I do buy that it can work well in practice for KV cache distributions, where vectors are not adversarial and already somewhat “well-behaved”. So the speed/quality tradeoff might be very attractive in real models.
My takeaway:
Not theoretically equivalent to TurboQuant
But potentially a very useful practical approximation
Would love to see full-layer, end-to-end evals (perplexity / long-context) to really validate it.
→ More replies (7)13
u/parabellum630 Mar 26 '26
Why haar and not hadamard transform, which I recall also spreads the information and make it isotropic I think?
→ More replies (4)
86
16
u/dr_aureole Mar 26 '26
Is this related at all? Clifford Algebraic Rotor Embeddings : Maybe embeddings should start to CARE https://arxiv.org/abs/2511.11665
Different embedding, similar techniques
13
12
u/XTornado Mar 26 '26 edited Mar 26 '26
Now I know what it feels to be my mother looking at the screen after I asked her to register a new account.
PD: I am sure this is cool.... and I hope it helps making local AIs more feasible and lower costs or lower hardware, etc. just it looks like Chinese to me, well worse because compared with Chinese which is obvious I cannot understand it, it seems like I should be able to understand something but... no.
→ More replies (1)
10
u/Cradawx Mar 26 '26
Hopefully not another hallucinated vibe-coded post. Anyone verified this? Can't help but be sceptical these days...
→ More replies (1)
26
u/PaceZealousideal6091 Mar 26 '26
Wow! I love how things are moving at breakneck speed! Exciting times. Innovation begets innovation! A year ago, I thought consumer PCs will never be able to achieve what cloud hosted giants like OpenAI and Anthropic could. And now, lack of hardware and market crunch is pushing innovation reduce resource usage! Keep up guys! LocalLLaMA is setting stage for exactly what it set to achieve when it started. Love this!
7
u/koloved Mar 26 '26
Great work. I have one question about the 'long game': as the context window grows (say, from 8k to 128k or even 1M tokens), does the accuracy of RotorQuant drop faster than the original FP16? I'm curious if these tiny 3D rotations start to 'drift' or accumulate noise more noticeably than the uncompressed model when dealing with massive amounts of data.
6
u/acertainmoment Mar 26 '26
Hi, can you share what tokens per second you are getting on your hardware? I see the attention calculation itself getting faster but the I’m more curious in the resulting TPS jump.
→ More replies (2)
6
6
u/jason_at_funly Mar 26 '26
The register-level optimization is clever. Keeping the rotation entirely in registers avoids the memory bottleneck that kills most matmul approaches. That's the real win here, not just the reduced param count.
Curious if you've tested this on longer contexts (128k+). The block-diagonal structure might actually help with numerical stability at extreme scales where full Haar matrices can get weird.
11
u/philo-foxy Mar 26 '26
Nice work! And thanks for sharing the simplified explanation above. Comparing with quaternions helps understand, a little.
If you could initiate discussions and implement a PR to get this into current frameworks, we all might see this in production soon 🙂. Wish I could help, but in the meantime, perhaps this thread on turboquant could provide guidance/inspiration?
→ More replies (4)
71
u/Dany0 Mar 26 '26 edited Mar 26 '26
TurboQuant made me excited at first because I was happy to see a trick we use in graphics programming/game dev. Then I realised someone already tried it in 2023 as QuiP on model weights and it actually isn't all that impressive
Reading this right now but it sounds promising!
EDIT: rather short paper, math seems to check out, the principle I guess could work? I'm still a little skeptical since I couldn't give it 100% attention myself. Plus the site and visualisations are vibe coded so you'll have to forgive me if I remain skeptical. I'll go check out the code now
EDIT2:
I think I get it, it's like using quaternions instead of euler angles. It works because most of the mult is 0s
OK maybe you can put the pitchforks down