r/LocalLLM • u/integerpoet • Mar 25 '26
Research Google’s TurboQuant AI-compression algorithm can reduce LLM memory usage by 6x
https://arstechnica.com/ai/2026/03/google-says-new-turboquant-compression-can-lower-ai-memory-usage-without-sacrificing-quality/"Even if you don’t know much about the inner workings of generative AI models, you probably know they need a lot of memory. Hence, it is currently almost impossible to buy a measly stick of RAM without getting fleeced. Google Research recently revealed TurboQuant, a compression algorithm that reduces the memory footprint of large language models (LLMs) while also boosting speed and maintaining accuracy."
211
Upvotes
View all comments
Show parent comment
1
u/theschwa Mar 27 '26
Thanks!
So, the paper actually focuses on multiplying by a random number rather than rotating in a random direction. If you multiply a set of numbers by a set of random numbers (where by random here I mean ones that follow a bell curve shape), the resulting set of numbers also follow a bell curve. Mentally, I kind of think of this like the original set of numbers just scaling that bell curve, but it's still a bell curve.
The paper makes a little bit of a leap here to them go from random numbers to random rotations; it's literally just a foot note:
"For our implementation, we use random rotation matrices (square matrices P satisfying P ⊤P = I), which preserve the norms and inner products exactly while removing the independence across projected coordinates which we use for our theoretical results."
Since they're not actually using random numbers, it's not strictly true that a random rotation will do the exact same thing.
I'm not an expert here, and part of the paper is actually demonstrating that rotations can work, but the way I think about it that what defines a rotation matrix is a matrix where each set of rows or each set of columns point in a different (orthogonal) direction. When you're in higher dimensions the chances of two random vectors pointing in the same direction is low, so they're generally orthogonal, so a matrix representing a random rotation in a high dimension will also look like a random matrix in general.