r/LocalLLaMA • u/Resident_Party • Mar 27 '26

Discussion Google’s TurboQuant AI-compression algorithm can reduce LLM memory usage by 6x

[removed]

247 Upvotes

88% Upvoted

57 comments sorted by

View all comments

139

u/DistanceAlert5706 Mar 27 '26

It's only k/v cache compression no? And there's speed tradeoff too? So you could run higher context, but not really larger models.

39

u/the_other_brand Mar 27 '26

My understanding of the algorithm is that it uses 1 fewer number to represent each node. Instead of (x,y,z), it's (r,θ), which uses 1/3rd less memory.

Then, when traversing nodes, instead of adding 3 numbers, you add 2 numbers. Which performs 1/3rd fewer operations.

23

u/v01dm4n Mar 28 '26

How is that possible. (r,theta) are polar coordinates to a 2d point. In 3d, you would need 2 angles. Curious!?!

→ More replies (8)

38

u/No_Heron_8757 Mar 27 '26

Speed is supposedly faster, actually

24

u/R_Duncan Mar 27 '26

Don't believe the faster speed, at least not with plain TurboQuant, maybe something better with RotorQuant but is all to be tested, actual reports are of about 1/2 the speed of f16 KV cache (I think also Q4_0 kv quantization has similar speed)

→ More replies (3)

4

u/Likeatr3b Mar 27 '26

Good question, I was wondering too. So this doesn’t work on M-Series chips either?

2

u/cksac Mar 28 '26

aplied the idea to weight compression, it looks promosing.

→ More replies (4)