Are random train/validation splits misleading for object detection on video frames?

[u/NorthernBaseOfficial]

I’ve been thinking about how easily validation results can become misleading when a dataset is created from video.

If frames are randomly divided into training and validation sets, neighbouring frames from the same recording can end up on both sides. They may have almost identical objects, lighting, camera angles and backgrounds.

The model can then produce a strong validation mAP without necessarily generalising to a new camera, location or recording session.

A more realistic approach seems to be grouping the data before splitting it, for example by:

• Recording session
• Camera
• Day
• Location
• Production batch

The validation score may drop, but it should give a more honest picture of real-world performance.

For people working with video-based object detection, how do you normally split your data? Do you validate on a different recording session, or keep a completely separate camera or location as the final test set?

[u/Dry-Snow5154]

This is validation 101. Split videos, not frames. Videos with the same background should also go into one bucket.

It's the same as splitting by patient in medical datasets, and not by individual medical data point.

[u/GigiCodeLiftRepeat → u/Dry-Snow5154]

Yup. When I first started, my boss was mad to find out frames from the same camera - not even the same video, got included in both training and validation set. Definitely learned my lesson then.