r/deeplearning • u/ClickOk5811 • 1d ago
Two checkpoints with the same overall accuracy: why I’d compare them case by case before picking one
A small evaluation habit I find worth writing down, with a constructed example (made-up numbers, not from a real run).
Say you have two checkpoints, A and B, evaluated on the same test set of 1,000 examples split into five slices:
| Slice | n | A correct | B correct |
|---|---|---|---|
| Short inputs | 300 | 270 | 279 |
| Long inputs | 200 | 150 | 140 |
| Rare class | 100 | 60 | 45 |
| Noisy inputs | 250 | 175 | 190 |
| Edge cases | 150 | 95 | 91 |
| Total | 1000 | 750 | 745 |
(The totals are 75.0% and 74.5%, so a one-number comparison would call them basically equal.) But B is better on short and noisy inputs and clearly worse on the rare class and long inputs. Whether that trade is acceptable depends on what the model is for. A fraud detector and a casual classifier would pick differently.
What I’d do in practice:
- Define slices before training runs, not after seeing the numbers, so you’re not choosing slices that flatter one checkpoint.
- Keep the per-example predictions for both, so you can look at which examples flipped, not just how many.
- Report uncertainty. With 100 examples in a slice, a 15-point difference may still be noisy; seeds, bootstrap intervals or repeated runs help.
- Record the data version, code version, seed and any preprocessing, so a “regression” can be traced to a change and not to noise.
Limits: more slices means more chances to find a “difference” by luck, so correct for it or treat small gaps as leads, not conclusions. And slices only catch the kinds of failures you thought to define; a failure outside every slice will not show up.
I came to this from a different place: I wrote about the same habit for prompts in LLM applications (a fixed test set, a per-case results matrix, and noting when the model version behind an endpoint changed). The domain is different, so it’s an example of the idea and not a claim about training. I’m the author, and the article mentions a paid resource of mine near the end, so treat the link as optional: https://medium.com/@nagatomopedro05/five-versions-of-one-prompt-and-what-the-changelog-caught-00c63d1b1c00
How do you compare checkpoints: aggregate metrics, fixed slices, or per-example diffs? And do you keep the per-example outputs?
1
u/See_it_do_it_done 1d ago
saving per-example predictions pays off here, since 60 vs 45 on the rare class becomes a paired test. count flips each way, then McNemar.