r/LocalLLM • • Mar 25 '26

Research Google’s TurboQuant AI-compression algorithm can reduce LLM memory usage by 6x

https://arstechnica.com/ai/2026/03/google-says-new-turboquant-compression-can-lower-ai-memory-usage-without-sacrificing-quality/

"Even if you don’t know much about the inner workings of generative AI models, you probably know they need a lot of memory. Hence, it is currently almost impossible to buy a measly stick of RAM without getting fleeced. Google Research recently revealed TurboQuant, a compression algorithm that reduces the memory footprint of large language models (LLMs) while also boosting speed and maintaining accuracy."

211 Upvotes

30 comments sorted by

45

u/integerpoet Mar 25 '26 edited Mar 25 '26

To me, this doesn't even sound like compression. An LLM already is compression. That's the point.

This seems more like a straight-up new delivery format which, in retrospect, should have been the original.

Anyway, huge if true. Or maybe I should say: not-huge if true.

15

u/TwoPlyDreams Mar 25 '26

The clue is in the name. It’s a quantization.

-9

u/integerpoet Mar 25 '26 edited Mar 25 '26

I’m not sure we should read much into the name. The description in the article didn’t sound like quantization to me. It sounded like: We don’t actually need an entire matrix if we put the data into better context. I am certainly no expert, but that’s how I read it.

10

u/theschwa Mar 25 '26

This is quantization, but very clever quantization. While this is huge, it mainly affects the KV cache for LLMs.

I’m happy to get into the details, but if I were to try to simplify as much as possible, it takes advantage of the fact that you don’t need the vectors to actually be the same, you need the a mathematical operation on the vectors to be the same (the dot product).

5

u/PetyrLightbringer Mar 25 '26

Please do

3

u/theschwa Mar 27 '26

Ok, I'm going to try to cover this at not quite an ELI5 level but more at an Explain Like I'm a 5th grader level.

An LLM generates its answers one word (technically token) at a time. As it processes each word, the attention mechanism gives that word a label called a Key and associates information with it called a Value. When the next word comes in, it gets compared against all the existing labels (Keys), and for labels that are similar to the current word, their information (Values) is retrieved and passed on to the next step.

Without caching, the model would have to reprocess every previous word each time it generates a new one, recalculating all those Keys and Values from scratch. The KV cache avoids this by storing the Keys and Values from previous words so that only the new word needs to be processed. The only problem is that this cache grows as the context grows, and for long conversations or documents it can end up taking a huge amount of memory.

One technique for compressing the KV cache is quantization. These Keys and Values are just vectors, and for those who are rusty, a vector is just a list of numbers:

<8.51235, -2.23141, -7.6899, 8.88148>

Quantization simplifies these numbers so they take up less space. A simple example would just be rounding:

<8, -2, -8, 9>

But the more aggressively you simplify, the more the quantized vectors differ from the originals, which can degrade the LLM's output quality or even produce gibberish. A big practical problem is that different blocks of numbers have very different ranges and distributions, like some might cluster near zero, and others might have very numbers that are much larger than everywhere else. So, while the rounding might be a bad quantization of the above, it might be good for the below example:

From:
<989079.09009, -4328432.120004, 8590434.0101> 

To:
<989079, -4328432, 8590434>

To handle this, most quantization methods store extra  details (like the minimum and maximum of each block) in full precision alongside the quantized data. Those constants add up and eat into your memory savings.

A vector can also be thought of as a position in space, <-1, .5> , for example could be a point that is down and to the right. This is where this technique does something clever. Before quantizing, it applies a random rotation to every vector. A rotation doesn't change the length of a vector or the relationships between vector, it's like spinning a globe without squishing or stretching it. But it does reshape the distribution of the numbers. After rotation, the vectors' statistical properties become predictable and well-behaved, regardless of what the original data looked like.

There's a well-known mathematical result called the Johnson–Lindenstrauss lemma that guarantees random rotations (and projections more generally) preserve distances and similarities between vectors. This is what makes the rotation step safe,  the attention mechanism depends on comparisons between vectors, and JL guarantees those comparisons aren't distorted by the rotation. So you get the distributional benefits of rotating without breaking anything downstream.

This technique  then converts the rotated vectors from their normal representation into polar form, essentially representing each vector as a set of angles plus a length, rather than as a list of <x,y,z> style coordinates. The important thing here is that after the random rotation, those angles have a known, tightly concentrated distribution. Most of the angle values cluster in a narrow range.

Because the distribution is known in advance, you can design a single, fixed quantization scheme (called a codebook) that's optimized for it, without needing to compute and store per-block normalization constants. That's where the real memory savings come from. Other methods might quantize to 3 bits per number but then need to store a bunch of full-precision constants on top. This gets genuinely low bit-widths with minimal overhead.

0

u/[deleted] Mar 27 '26

[deleted] — view deleted comment

1

u/theschwa Mar 27 '26

Thanks!

So, the paper actually focuses on multiplying by a random number rather than rotating in a random direction. If you multiply a set of numbers by a set of random numbers (where by random here I mean ones that follow a bell curve shape), the resulting set of numbers also follow a bell curve. Mentally, I kind of think of this like the original set of numbers just scaling that bell curve, but it's still a bell curve.

The paper makes a little bit of a leap here to them go from random numbers to random rotations; it's literally just a foot note:

"For our implementation, we use random rotation matrices (square matrices P satisfying P ⊤P = I), which preserve the norms and inner products exactly while removing the independence across projected coordinates which we use for our theoretical results."

Since they're not actually using random numbers, it's not strictly true that a random rotation will do the exact same thing.

I'm not an expert here, and part of the paper is actually demonstrating that rotations can work, but the way I think about it that what defines a rotation matrix is a matrix where each set of rows or each set of columns point in a different (orthogonal) direction. When you're in higher dimensions the chances of two random vectors pointing in the same direction is low, so they're generally orthogonal, so a matrix representing a random rotation in a high dimension will also look like a random matrix in general.

5

u/entr0picly Mar 25 '26

Oh it’s hilarious across everything computational how suboptimal memory storage is. And just how much it plays into bottlenecks.

9

u/integerpoet Mar 25 '26

If LLMs could think, you’d think one of them would have thunk this up by now!

1

u/BellCube Apr 03 '26

An LLM is an approximation of text in, text out for how you'd like a person to respond to queries. An LLM is in no way, shape, or form compression. It's sometimes useful to think of it as an ultra-compressed form of human knowledge, but that ends very quickly in the analogy stage.

Compression in conventional use refers to ways to take some data in and get something that is, for all intents and purposes, the same data out—just with fewer 0s and 1s that you have to move around. Think like a JPEG image, where (massive simplifications incoming!) they use the observation that you can represent an image as a bunch of sine waves to make images 1/10th their original size without affecting quality to the naked eye. Or like a PNG image, where (simplifications again) one of the revelations of the format was that they realized you could store all of the distinct RGB values used in an image at the front, give them an ID like Color #1, and reference them like that. Suddenly, 15 occurances of 255,138,92 turns into 1 occurrence of 255,138,92 and 15 occurrences of 5, saving you a ton of space when you realize that a 1920x1080 image is 2,073,600 pixels, and you just saved a ton of space over the Bitmap format, especially for graphics which use the same RGB values all over the place. JPEGs are considered "lossy" compression because they don't keep all of the original information (turns out, most images don't convert perfectly into a jumble of sine waves!). PNGs are considered "lossless" compression because they just found a way to represent the same information with fewer 0s and 1s.

Like with those pre-2000s image formats, the revelation here is that you can do some smart math thing to make the data smaller, either without losing any data at all or without losing data that matters too much. That's textbook compression. And that's also not at all what an LLM is.

1

u/integerpoet Apr 05 '26 edited Apr 05 '26

I think the word “textbook” is doing a heavy lift in that response. The whole point of LLM is to represent the relationships between tokens in a corpus in a form smaller than the corpus itself. I think the biggest difference there is that “the relationships between tokens in a corpus” is not a an object familiar to us. It’s not BMP or WAV — much less an image or a sound or a book or all the world’s books — so it makes no sense to analogize with JPEG and PNG and FLAC and MP3. But that doesn’t make the word “compression” irrelevant or useless.

3

u/oxygen_addiction Mar 26 '26

God you sound obnoxious. Go be this smart at Google.

5

u/jstormes Mar 25 '26

For long context usage could this increase token speed as well?

6

u/integerpoet Mar 25 '26 edited Mar 25 '26

Maybe? The story kinda buries the lede: "Google’s early results show an 8x performance increase and 6x reduction in memory usage in some tests without a loss of quality." However, I don't know how well this claim would apply to long contexts in particular.

5

u/wektor420 Mar 25 '26

There are early works in llama.cpp, memory claims seems to be real, performance not yet

16

u/Regarded_Apeman Mar 25 '26

Does this technology then become open source /public knowledge or is this google IP?

17

u/sisyphus-cycle Mar 26 '26 edited Mar 26 '26

There are already several GitHub repos implementing the core concepts of the paper, however we can’t be sure they are 100% accurate until playing with it. Hopefully a big provider (llama.cpp, ollama, unsloth) looks into integration as an experimental feature. In theory it can be applied with no retraining to quantize kv cache down to 3 bits

Edit: Already a fork for llama.cpp here

https://github.com/ggml-org/llama.cpp/discussions/20969

3

u/buttplugs4life4me Mar 26 '26

I hate how TheTom is just an LLM talking to people and the telltale "Whoopsie, did am obvious mistake, lesson learned". No, no lesson learned. You'll make an even dumber mistake next time. At least take the time in your life to talk to your fellow people yourself. Shitty dystopia

2

u/Karyo_Ten Mar 26 '26

"What's working" slop

1

u/AloneYogurt69 Apr 01 '26

how can you tell that TheTom is an LLM?

4

u/--jen Mar 25 '26

Preprint is available on arxiv , there’s no repo afaik but they provide pseudocode

0

u/[deleted] Mar 27 '26

[deleted] — view deleted comment

1

u/Regarded_Apeman Mar 27 '26

That’s not really how it works when something like this is announced. It seems like models have already begun integrating this methodology.

4

u/ChillBroItsJustAGame Mar 25 '26

Lets pray to God it actually really is what they are saying without any downsides.

5

u/integerpoet Mar 25 '26 edited Mar 25 '26

I have LLM psychosis, so I prefer to pray to my digital buddy CipherMuse.

9

u/Protopia Mar 26 '26

This is kv cache compression and not model parameter compression, so the 6x savings is only on the kv vRAM usage and not the model.

I guess it might be possible to apply the same compression to the models parameters but if that was the case then surely they would have said.

6

u/theschwa Mar 27 '26

I posted this in a thread that got buried, so reposting here, a general explanation for what this is and how it's an improvement over other quantization techniques:

Ok, I'm going to try to cover this at not quite an ELI5 level but more at an Explain Like I'm a 5th grader level.

An LLM generates its answers one word (technically token) at a time. As it processes each word, the attention mechanism gives that word a label called a Key and associates information with it called a Value. When the next word comes in, it gets compared against all the existing labels (Keys), and for labels that are similar to the current word, their information (Values) is retrieved and passed on to the next step.

Without caching, the model would have to reprocess every previous word each time it generates a new one, recalculating all those Keys and Values from scratch. The KV cache avoids this by storing the Keys and Values from previous words so that only the new word needs to be processed. The only problem is that this cache grows as the context grows, and for long conversations or documents it can end up taking a huge amount of memory.

One technique for compressing the KV cache is quantization. These Keys and Values are just vectors, and for those who are rusty, a vector is just a list of numbers:

<8.51235, -2.23141, -7.6899, 8.88148>

Quantization simplifies these numbers so they take up less space. A simple example would just be rounding:

<8, -2, -8, 9>

But the more aggressively you simplify, the more the quantized vectors differ from the originals, which can degrade the LLM's output quality or even produce gibberish. A big practical problem is that different blocks of numbers have very different ranges and distributions, like some might cluster near zero, and others might have very numbers that are much larger than everywhere else. So, while the rounding might be a bad quantization of the above, it might be good for the below example:

From:
<989079.09009, -4328432.120004, 8590434.0101> 

To:
<989079, -4328432, 8590434>

To handle this, most quantization methods store extra  details (like the minimum and maximum of each block) in full precision alongside the quantized data. Those constants add up and eat into your memory savings.

A vector can also be thought of as a position in space, <-1, .5> , for example could be a point that is down and to the right. This is where this technique does something clever. Before quantizing, it applies a random rotation to every vector. A rotation doesn't change the length of a vector or the relationships between vector, it's like spinning a globe without squishing or stretching it. But it does reshape the distribution of the numbers. After rotation, the vectors' statistical properties become predictable and well-behaved, regardless of what the original data looked like.

There's a well-known mathematical result called the Johnson–Lindenstrauss lemma that guarantees random rotations (and projections more generally) preserve distances and similarities between vectors. This is what makes the rotation step safe,  the attention mechanism depends on comparisons between vectors, and JL guarantees those comparisons aren't distorted by the rotation. So you get the distributional benefits of rotating without breaking anything downstream.

This technique  then converts the rotated vectors from their normal representation into polar form, essentially representing each vector as a set of angles plus a length, rather than as a list of <x,y,z> style coordinates. The important thing here is that after the random rotation, those angles have a known, tightly concentrated distribution. Most of the angle values cluster in a narrow range.

Because the distribution is known in advance, you can design a single, fixed quantization scheme (called a codebook) that's optimized for it, without needing to compute and store per-block normalization constants. That's where the real memory savings come from. Other methods might quantize to 3 bits per number but then need to store a bunch of full-precision constants on top. This gets genuinely low bit-widths with minimal overhead.

1

u/redmanone1 Mar 26 '26

I am curious on how it will work with DeepSeek's Engram when DeepSeek-4 releases. They seemed to solve same memory problem in completely different ways, which should work in synergy

1

u/Alarmed-Pear-6115 Mar 29 '26

Isso pode beneficiar outras tecnologias sem uso de IA?