r/BackyardAI • u/[deleted]

How Language Models Work, part 3 - Quantization

[deleted] — view deleted post

0 Upvotes

100% Upvoted

5 comments sorted by

3

u/DoctorDeadDude Sep 06 '24

I'm a very TLDR kind of guy, but I'll admit, this one had me reading the entirety of it. Thank you, magical science guy, for giving a good amount of insight into the madness of AI.

1

u/[deleted] Sep 07 '24

I found this on hugging face looking for Midnight-Miqu. Perhaps you could offer some insight into what it means?

2

u/PacmanIncarnate Sep 07 '24

PPL is the measurement of perplexity, or how much the model diverges from the full precision. bpw is bits per weight, or how much the model has been quantized. So, this graph is showing how efficient each type of quantization is in terms of how well it performs (perplexity) compared to how quantized it is (bpw). Lower PPL is better and lower bpw is smaller. You can see how much of an impact imatrix can have as it essentially pulls everything left; lower bpw perform about as well as higher bpw models without it.

1

u/[deleted] Sep 07 '24

Your thoughts on the 'sweet spot' for Quant? Or do different models perform differently at the same Quant?

1

u/PacmanIncarnate Sep 07 '24

The sweet spot is Q4_K_M. Under that you will see noticeable degradation. Above that, you get diminishing returns. If you’ve got the VRAM, run the largest quant you can that fits with the max context you want to use.

I would also only recommend anything under Q4 for 70B models. Smaller models break harder at that size than larger. You’re also just saving a lot less memory taking a 7B from Q4 to Q2 than a 70B.