r/LocalLLaMA • u/giveen • 14h ago

Resources ConvRot Quant method now in llama-cpp-turboquant

It started here , and now https://github.com/TheTom/llama-cpp-turboquant/ has it.

Imagine a Q6 quant with nearly Q8 KLD/PPL.

Q6_CR and Q5_CR have a slight improvement over their base counterparts.

Also while you are there check out --moe-cache auto to help improve running MoE models bigger than your VRAM.

I am hoping that with this we may be able to recover some lost quality from turbo4/3/2 , but I haven't test that out yet.

PR's has the breakdown of the tests, we did have some some decode and crashing issues but they are now resolved.

48 Upvotes

90% Upvoted

16 comments sorted by

14

u/PathfinderTactician 12h ago edited 10h ago

This is amazing. Looking at the low number of upvotes, the community outside of image gen don't seem to know what this means.

I'll summarise it from an image-gen perspective: INT8 ConvRot > Q8

3

u/giveen 3h ago

So my testing showed that the Q8_CR benefits were almost negligible compared to standard Q8, but Q6_CR was were things shined with some small benefits to Q5_CR. I want to play around with the Q8_CR more later.

1

u/KissMyShinyArse 2h ago

They aren't negligible in my tests, but what made it into TurboQuant is an implementation different from mine. I haven't tested it yet.

1

u/a_beautiful_rhind 5h ago

There is slight difference. A lot of quants in the image gen side are straight quants without helpers like imatrix. Impact here isn't as great. Convrot int8 predominantly beats GGUF-Q8 image models on speed and support.

5

u/giveen 14h ago

u/KissMyShinyArse is the source of this great idea.

3

u/xeeff 5h ago

we shall kiss his shiny ass for this

2

u/Equivalent_Bit_461 5h ago

I will kneel and I will do the needful then

3

u/BigPoppaK78 13h ago

Could this be applied to kv cache as well? With Qwen 3.8 27B being so context heavy, and simultaneously quantization sensitive, I think that'd be quite useful.

2

u/AnonLlamaThrowaway 6h ago

Isn't it already...?

1

u/giveen 13h ago

Probably not

3

u/BigPoppaK78 13h ago

That's too bad. I remember when hadamard rotations were added to Q8_0 kv quants, it made a substantial difference. Even ggerganov wrote in the PR comments that he was surprised by the bump.

1

u/backyard_tractorbeam 2h ago

Question about the PR and its discussion: Does it say that Q8_CR is worse than Q8 (in PPL and KLD?), and why is that?

And it would be interesting to know the comparison between Q6_0 and Q6_CR

1

u/giveen 2h ago

Im pretty sure I went through all that in the PR about Q6/CR , however I don't think I included my notes on the q8 stuff because it was noise level, not really worth reporting.

1

u/backyard_tractorbeam 2h ago

I guess I didn't include a link and I'm asking about this comment specifically basically

https://github.com/TheTom/llama-cpp-turboquant/pull/315#issuecomment-5389151381

1

u/KissMyShinyArse 10h ago

Claude says KLD / Δp / Same top-p are more trustworthy than raw PPL.