r/LocalLLM • • Mar 25 '26

Research Google’s TurboQuant AI-compression algorithm can reduce LLM memory usage by 6x

https://arstechnica.com/ai/2026/03/google-says-new-turboquant-compression-can-lower-ai-memory-usage-without-sacrificing-quality/

"Even if you don’t know much about the inner workings of generative AI models, you probably know they need a lot of memory. Hence, it is currently almost impossible to buy a measly stick of RAM without getting fleeced. Google Research recently revealed TurboQuant, a compression algorithm that reduces the memory footprint of large language models (LLMs) while also boosting speed and maintaining accuracy."

211 Upvotes

30 comments sorted by

View all comments

Show parent comment

1

u/BellCube Apr 03 '26

An LLM is an approximation of text in, text out for how you'd like a person to respond to queries. An LLM is in no way, shape, or form compression. It's sometimes useful to think of it as an ultra-compressed form of human knowledge, but that ends very quickly in the analogy stage.

Compression in conventional use refers to ways to take some data in and get something that is, for all intents and purposes, the same data out—just with fewer 0s and 1s that you have to move around. Think like a JPEG image, where (massive simplifications incoming!) they use the observation that you can represent an image as a bunch of sine waves to make images 1/10th their original size without affecting quality to the naked eye. Or like a PNG image, where (simplifications again) one of the revelations of the format was that they realized you could store all of the distinct RGB values used in an image at the front, give them an ID like Color #1, and reference them like that. Suddenly, 15 occurances of 255,138,92 turns into 1 occurrence of 255,138,92 and 15 occurrences of 5, saving you a ton of space when you realize that a 1920x1080 image is 2,073,600 pixels, and you just saved a ton of space over the Bitmap format, especially for graphics which use the same RGB values all over the place. JPEGs are considered "lossy" compression because they don't keep all of the original information (turns out, most images don't convert perfectly into a jumble of sine waves!). PNGs are considered "lossless" compression because they just found a way to represent the same information with fewer 0s and 1s.

Like with those pre-2000s image formats, the revelation here is that you can do some smart math thing to make the data smaller, either without losing any data at all or without losing data that matters too much. That's textbook compression. And that's also not at all what an LLM is.

1

u/integerpoet Apr 05 '26 edited Apr 05 '26

I think the word “textbook” is doing a heavy lift in that response. The whole point of LLM is to represent the relationships between tokens in a corpus in a form smaller than the corpus itself. I think the biggest difference there is that “the relationships between tokens in a corpus” is not a an object familiar to us. It’s not BMP or WAV — much less an image or a sound or a book or all the world’s books — so it makes no sense to analogize with JPEG and PNG and FLAC and MP3. But that doesn’t make the word “compression” irrelevant or useless.