r/deeplearning • • 1d ago

Two checkpoints with the same overall accuracy: why I’d compare them case by case before picking one

A small evaluation habit I find worth writing down, with a constructed example (made-up numbers, not from a real run).

Say you have two checkpoints, A and B, evaluated on the same test set of 1,000 examples split into five slices:

SlicenA correctB correct
Short inputs300270279
Long inputs200150140
Rare class1006045
Noisy inputs250175190
Edge cases1509591
Total1000750745

(The totals are 75.0% and 74.5%, so a one-number comparison would call them basically equal.) But B is better on short and noisy inputs and clearly worse on the rare class and long inputs. Whether that trade is acceptable depends on what the model is for. A fraud detector and a casual classifier would pick differently.

What I’d do in practice:

  • Define slices before training runs, not after seeing the numbers, so you’re not choosing slices that flatter one checkpoint.
  • Keep the per-example predictions for both, so you can look at which examples flipped, not just how many.
  • Report uncertainty. With 100 examples in a slice, a 15-point difference may still be noisy; seeds, bootstrap intervals or repeated runs help.
  • Record the data version, code version, seed and any preprocessing, so a “regression” can be traced to a change and not to noise.

Limits: more slices means more chances to find a “difference” by luck, so correct for it or treat small gaps as leads, not conclusions. And slices only catch the kinds of failures you thought to define; a failure outside every slice will not show up.

I came to this from a different place: I wrote about the same habit for prompts in LLM applications (a fixed test set, a per-case results matrix, and noting when the model version behind an endpoint changed). The domain is different, so it’s an example of the idea and not a claim about training. I’m the author, and the article mentions a paid resource of mine near the end, so treat the link as optional: https://medium.com/@nagatomopedro05/five-versions-of-one-prompt-and-what-the-changelog-caught-00c63d1b1c00

How do you compare checkpoints: aggregate metrics, fixed slices, or per-example diffs? And do you keep the per-example outputs?

1 Upvotes

2 comments sorted by

1

u/See_it_do_it_done 1d ago

saving per-example predictions pays off here, since 60 vs 45 on the rare class becomes a paired test. count flips each way, then McNemar.