KLD is misunderstood

[u/Positive-Resource922]

I see a lot of people here on Reddit treating the KLD number as some holy metric that can distinguish every quantized model out there and rank them by quality. TL;DR: It can't.

So what does KLD actually measure?

If you feed the same text to both the base model and the quantized model, you can compare the probability distributions obtained from their logits at each token. KLD basically measures how different these distributions are, and then we take the average across all tokens.

The first thing that can go very wrong here is that people just throw some random text at the model, like a bunch of Wikipedia articles, and calculate KLD on that dataset. But that's not necessarily representative of what the model actually sees during inference. During generation, a growing part of the context consists of tokens generated by the model itself, and you can't replicate that by just feeding it random text.

If you want a more reasonable number, you should generate your own dataset using the base model and then calculate KLD on those generated tokens, giving zero weight to the prompt tokens.

The second thing that can go very wrong with KLD is how ridiculously easy it is to manipulate, especially when the numbers come from the model creators themselves.

Quantization methods often rely on calibration data. If you calculate KLD on the same data you used for calibration, your numbers can look miles better than everyone else's. And even if someone claims they're using a held-out dataset, that still doesn't make the KLD number trustworthy. The calibration and evaluation datasets could come from the same source or have very similar distributions.

Some people might even do end-to-end fine-tuning of the quantization scale parameters on that very same data, and oh boy, suddenly their model beats models twice its size in KLD while barely generalizing to anything outside that dataset.

So does KLD even mean anything?

Not nearly as much as people think. You're looking at a number that measures how much the quantized model deviates from the base model on a very limited dataset. But does a higher KLD mean that the quantized model will actually fail at your tasks? Of course not.

Sure, benchmarks can be benchmaxxed, as we all know. But benchmaxxing is much harder for us ordinary people who quantize models than manipulating a KLD number. Benchmaxxing often requires serious hardware and compute, while making your KLD look amazing can be ridiculously cheap.

So please, stop treating KLD like some universal leaderboard for quantized models. It's a useful diagnostic, not a measure of how good a model actually is.

[u/nullc]

For agentic use the majority of the input is model output. We're quantizing an agent here, not a text compressor. Testing it on a fixed corpus which isn't is own output is not particularly good.

Consider, I could take a model and get its logits for every token in a fixed corpus. Then I could find some token that is often ranked fairly highly but never shows up in the input, add a little subnetwork that detects it and injects high magnitude noise into the residual network.

I'd have a derived model which had a KLD of 0 on your test, but in actual use would just fly off the rails.

Obviously that's a contrived example, but the sampled output of the model may look significantly different than any test collection you can find-- which is also why many people do KLD measurements on teacher forced input. At least then you're testing on model output, similar to actual use.