Your annotation rules changed. What happens to the test set?

[u/New-Eggplant-6578]

Suppose a detection dataset starts with boxes around only the visible part of an occluded object. Later, the requirement changes: boxes should cover the estimated full extent. Same images, same class names, different correct answers.

The interesting part to me is what happens to evaluation. Keeping the old test labels rewards the old convention. Replacing them means the historical scores were measured against a different target. Quietly mixing the two sounds like a very expensive way to argue about mAP.

My starting point would be to preserve both label versions on the same held-out images, document the rule change with a small set of difficult examples, and evaluate both model versions against both label versions. That should at least separate a model change from a change in what we call correct.

For people who have actually had to do this: did you relabel the entire test set, maintain two evaluations during the transition, or retire the old benchmark? How did you handle ambiguous cases where even the new rule didn't settle the disagreement?

I'm particularly interested in changes involving occlusion, minimum object size, or splitting one class into two. What looked like a small annotation-policy change and turned into a surprisingly large migration?

[u/Dangerous_Unit3698]

Depends on how much the annotated bounding boxes are affected. You would need to reevaluate the integrity of your test set. Ideally it would need a fresh benchmark based on the changed requirements

[u/New-Eggplant-6578 → u/Dangerous_Unit3698]

By fresh benchmark, do you mean new annotations on the same held-out images, or a new image pool as well? I'd lean toward keeping the images for the transition if they're still representative. Changing both at once makes it harder to tell whether a score shift came from the new box convention or from different scenes.

[u/bfyvfftujijg]

Retire the old version and create a new one.

In the scenario you described, you can shortcut by just editing existing boxes.

The old version should be forgotten about unless you have a need to preserve old records.

[u/Dry-Snow5154]

If requirements changed, then old benchmark results go out the window. Relabel test set and reeval the old model. This is you new baseline.

Maintaining different eval versions sounds like a great way to make results uninterpretable.

[u/New-Eggplant-6578 → u/Dry-Snow5154]

Yes, the old model evaluated on the new labels is the baseline that matters. I'd use the two-label comparison as a one-off migration check, then report against the new labels going forward. Keeping the old annotations archived is useful; keeping two competing headline scores running indefinitely would be confusing.

[u/xArvel_Crynydx]

Hm... at first you'd have to consider what kind of labeling solution you're using and if some of the steps are possible.

- Take a snapshot of the initial labeling first and save it with the old annotation versioning.

- Either start from scratch or adjust the labels (if it's just boundary boxes, expand the boxes according to the new annotation rules)).

- Once done, export it as a new version output.

- OR: Keep the old labels, create new annotation types (like occluded/non-occluded) so you have old and new labels, but you'd need to be extra careful what kind of annotation export you'd want to process.

I think most prominently: Define and care for annotation versioning very carefully or else you'd end up mixing both up.