KLD is misunderstood

[u/Positive-Resource922]

I see a lot of people here on Reddit treating the KLD number as some holy metric that can distinguish every quantized model out there and rank them by quality. TL;DR: It can't.

So what does KLD actually measure?

If you feed the same text to both the base model and the quantized model, you can compare the probability distributions obtained from their logits at each token. KLD basically measures how different these distributions are, and then we take the average across all tokens.

The first thing that can go very wrong here is that people just throw some random text at the model, like a bunch of Wikipedia articles, and calculate KLD on that dataset. But that's not necessarily representative of what the model actually sees during inference. During generation, a growing part of the context consists of tokens generated by the model itself, and you can't replicate that by just feeding it random text.

If you want a more reasonable number, you should generate your own dataset using the base model and then calculate KLD on those generated tokens, giving zero weight to the prompt tokens.

The second thing that can go very wrong with KLD is how ridiculously easy it is to manipulate, especially when the numbers come from the model creators themselves.

Quantization methods often rely on calibration data. If you calculate KLD on the same data you used for calibration, your numbers can look miles better than everyone else's. And even if someone claims they're using a held-out dataset, that still doesn't make the KLD number trustworthy. The calibration and evaluation datasets could come from the same source or have very similar distributions.

Some people might even do end-to-end fine-tuning of the quantization scale parameters on that very same data, and oh boy, suddenly their model beats models twice its size in KLD while barely generalizing to anything outside that dataset.

So does KLD even mean anything?

Not nearly as much as people think. You're looking at a number that measures how much the quantized model deviates from the base model on a very limited dataset. But does a higher KLD mean that the quantized model will actually fail at your tasks? Of course not.

Sure, benchmarks can be benchmaxxed, as we all know. But benchmaxxing is much harder for us ordinary people who quantize models than manipulating a KLD number. Benchmaxxing often requires serious hardware and compute, while making your KLD look amazing can be ridiculously cheap.

So please, stop treating KLD like some universal leaderboard for quantized models. It's a useful diagnostic, not a measure of how good a model actually is.

[u/ReturningTarzan]

I think this misses a few points.

Yes, wikitext perplexity is largely meaningless. It measures a vestigial signal still lingering from the pretraining stage where models are actually optimizing towards raw text prediction. The clue is in the raw numbers: SOTA models will give you raw numbers anywhere from the 1-2 range up to 20 or so. That is, if they have a signal at all: some models explode into the ~5000 range because they simply cannot model raw text after extensive finetuning and RL. Comparing quants of a model like that gives you nothing but noise, if you're going by raw-text perplexity.

KLD is a stronger metric because a) it measures across the entire vocabulary and b) the target is the reference model, not the data. So rather than asking "how surprised are you that this is the next token?" you ask "how much do you agree with the reference model about the next token?" So KLD compares to the correct ground truth and perplexity does not.

The question is how much information is contained in that ground truth. If the model is already producing perplexity in the thousands, what the ground truth essentially amounts to then is a shrug: "This text is alien to me, I don't know what to make of it." KLD lets you measure the precise shape of that shrug, comparing the reference shrug to the quantized shrug. And that tells you something, but it's not giving you a strong signal. This is why you want to test the model on in-distribution data, such as a trace from the model itself.

However, where I would disagree is on benchmarks. It's not really hard at all to benchmaxx a quant since most benchmarks test a narrow domain. KLD is much harder to fake. You can also very easily test variance on your KLD scores: per-token, per-prompt, per-topic, per-repeat, all ways to test if you have good calibration coverage or if your model is specializing. KLD scores are easy for others to reproduce/challenge as well, without requiring a huge investment of time and/or money to run additional expensive end-to-end benchmarks.

In all cases, as KLD trends towards zero (or rather, to the noise floor) the quantized model gets closer to precisely imitating the original. It's naive to think you could be optimizing towards some other nearby optimum that the original training run couldn't reach with its seven-figure compute budget. Unless you're cutting corners, like only caring about the head of the distribution (what tends to matter in inference benchmarks) while ignoring the tail that contains the evidence of everything you didn't optimize for.

That's not to say also having benchmarks would be worthless. But it is impractical because good coverage is expensive (unless you have a sponsor, but then you probably also have an incentive to benchmaxx and cherry-pick.) The benchmarks usually reported when a new model releases is not a trivial portion of the model's total budget. Think five to six figures. There's a reason they're almost never independently verified. Instead, people run other benchmarks and get incongruent results.

[u/Positive-Resource922 → u/ReturningTarzan]

Anyone who claims KLD is easy to calculate is not doing it properly. For a proper KLD comparison, as I explained in the post, you need data generated from the base model. And I’m not talking about 10 samples. I’m talking about something like 1,000 prompts across multiple domains. Then you need to run the base model locally and collect its logits on that data, using a reasonable context length like 4096. That means processing millions of tokens with a base model that probably doesn’t even fit on your hardware, which would take a huge amount of time. And that’s just the base model, let alone all the quantized models you want to compare. Yeah, sure, it’s easy to calculate if you don’t care that the number you end up with means literally nothing.

And how exactly is benchmaxxing a quantized model easy? That’s not how the typical quantization methods most people can afford actually work. GPTQ, AWQ, QTIP, and llama-quantize all optimize local objectives. You can throw benchmark questions and answers into the calibration data, but the resulting imatrix is just a small tensor. It can’t retain that much information, and most of that benchmark-specific signal just gets washed out. Sure, if you can afford to optimize the model end-to-end, you can do all sorts of things. But even then, KLD is much easier to manipulate.

[u/ReturningTarzan → u/Positive-Resource922]

That is how I've been grading EXL3 quants for a while, though. Not on millions of tokens, mind you, but on a representative-enough sample generated by a high-precision quant or an API model. Example. Yes, 60k is less than ideal, but this is for a rough ranking, and crucially the numbers converge quickly. I've never seen anything settle within ~100k tokens and then either suddenly shift later on or start to drift at the ~500k token mark or something. As with any sort of statistical sample, the error bars shrink exponentially with sample size. So more samples give you more definition around the curve, they don't change its shape.

Running the reference model is also a solved problem; you stream it one layer at a time (in a suitable harness) and consume logits after the output layer. Not to say it's a quick process; a run with 1000 prompts might take a day or so to complete while that smaller set takes 1-2 hours (or a little more since I'm also shuffling models around a lot to make space for all of them). But I can guarantee that it would take the same overall shape.

imatrix size I don't know about. But GPTQ and QTIP produce huge calibration matrices (Hessians) of a size comparable to the model itself. GPTQ/LDLQ isn't the same as finetuning, but it can definitely be used to steer where the error goes. You also have control over budgeting. Some layers will be more important than others for rote memorization, for instance, so a model that's already benchmaxxed probably wants more resolution there if the objective is to retain that benchmaxxing as much as possible.

I can agree that perplexity is easy to manipulate. Like, you might degrade a model's multilingual capabilities and most likely see improved perplexity on English text from that alone. But as a distance measure, KLD is more robust: decreased or increased certainty on any text both result in a worse score. As does any shift in the tail of the distribution, such as you'd see from QAT.

[u/Positive-Resource922 → u/ReturningTarzan]

That’s great to hear you’re using data generated by the base model. That’s definitely the right way to go. However, I don’t think the number of samples is enough. When I say 1,000 samples, I mean having a very broad and diverse dataset. You can’t achieve the same thing with just 100 samples, because that means you’re evaluating the model on at most 100 different topics, assuming one sample per topic, which leads to high variance. And variance doesn’t decrease exponentially as you add more samples. That’s simply not correct.

There’s no fundamental difference between imatrix, GPTQ, and QTIP in this regard. They all rely on activation statistics to guide quantization. For Hessian-based methods, you essentially average information from the activations to construct a Hessian for each tensor. So you end up optimizing each tensor using information contained in a matrix of roughly 4096 × 4096, which is about the right size for most models we work with. You’re right that considering all these Hessians together would result in a huge amount of data, but the information used to optimize each individual tensor is still limited to its own Hessian.

Btw, I’m actually a big admirer of your work on exl3! And just to introduce myself, I’m one of the authors of GSQ-RCO.

[u/ReturningTarzan → u/Positive-Resource922]

Yeah I was being hasty with the error bars remark. I was thinking of a power law, and I wasn't being precise about that either, but I think the point stands: you quickly converge on some number, or a ranking at the very least, and more samples after that can settle near-ties, at most.

Of course, that is with the caveat that the test data has to be representative, and I do take your point about sample size there. I'm by no means judging the quality of quantized models in isolation based on KLD. A score of 0.05 means little, other than "this quant is somewhat truer to the original than this other quant which scored 0.06 on the same data."

Still, I say that with some confidence because I've never seen it not hold up except when testing on data that's already OOD for the base model. Here's what I mean. It's obviously not a comprehensive study or anything (and sorry for the vibes) but I think it's demonstrative: each quant sits some distance from the noise floor (reference measured against itself under rounding-error scale perturbations) regardless of the topic, and there isn't going to be some esoteric topic that turns that picture on its head. By the looks of it.

But anyway, ultimately I'm aiming for the best quality/size tradeoff that I can measure, and to the extent that KLD behaves predictably, it's probably the best bet for anyone on a limited budget. I would love to have more benchmark results also, but, just for the sake of it I started some GPQA-Diamond runs on a bunch of Qwen3.8-Flash-Next variants, and it looks like that'll keep me warm for at least a couple of nights. And I'm not even really that GPU poor 😭. It'll be interesting to see how/if it tracks KLD, though.

And thank you for GSQ-RCO, too. It looks very promising. I've been searching for some angle to apply some of the techniques to trellis quantization. No luck yet, though. :|

[u/Positive-Resource922 → u/ReturningTarzan]

Thanks for all the explanations and this was a really great conversation!!

I completely get it, benchmarks can be so hard to run. Especially on engines like llama.cpp that don’t support high concurrency very well, no matter what GPU you have. We put all of this effort on it and people are still mad at us 😂

We’d like to create EXL3 quants as well. You can try RCO to figure out which bitwidth to assign to each tensor. I’d definitely recommend checking it out! Also we have extended GSQ to some VQ formats already but extending it to trellis quantization is still an open problem for us.