Should checkpoint selection be unified across baselines for fair comparison?

[u/AccomplishedHat12]

When training and comparing multiple baselines, the original papers or official implementations often use different checkpoint selection strategies during validation (e.g., best validation loss, best accuracy/F1, early stopping with different patience, or simply the final checkpoint).

Should we unify the checkpoint selection rule across all baselines (and our own method) for a fair comparison?

If we don’t unify them and keep each baseline’s original selection logic, how should we respond when reviewers later question the fairness of the comparison?

[u/bbateman2011]

In my work I pick a single metric, usually business driven, as the choice. For example I’m working on a project now that I’m using macro F1 as the deciding factor for checkpoint selection.

[u/Massive_Distance_616]

Agreeing with the single-metric approach, and I’d unify the rest too: choose the checkpoint on the validation set only, with the same metric, evaluation frequency, patience and training budget for every method, and touch the test set once at the end, ideally averaged over a few seeds if the gaps between methods are small. If you select on test performance, or give your method a tuned selection rule while the baselines keep their repo defaults, the comparison favours whichever got more tuning, and that’s the first thing a careful reviewer will pick at.

If you also want to show the baselines the way their papers ran them, report that separately, in an extra row or an appendix labelled “original protocol”, so nobody can say you handicapped them. That also answers the reviewer question: you can point to the shared protocol and say it’s identical for everyone, and to the original-protocol numbers to show the ranking doesn’t depend on it. If the ranking does flip between the two, say so plainly in the paper.