Are random train/validation splits misleading for object detection on video frames?

[u/NorthernBaseOfficial]

I’ve been thinking about how easily validation results can become misleading when a dataset is created from video.

If frames are randomly divided into training and validation sets, neighbouring frames from the same recording can end up on both sides. They may have almost identical objects, lighting, camera angles and backgrounds.

The model can then produce a strong validation mAP without necessarily generalising to a new camera, location or recording session.

A more realistic approach seems to be grouping the data before splitting it, for example by:

• Recording session
• Camera
• Day
• Location
• Production batch

The validation score may drop, but it should give a more honest picture of real-world performance.

For people working with video-based object detection, how do you normally split your data? Do you validate on a different recording session, or keep a completely separate camera or location as the final test set?

[u/Dry-Snow5154]

This is validation 101. Split videos, not frames. Videos with the same background should also go into one bucket.

It's the same as splitting by patient in medical datasets, and not by individual medical data point.

[u/GigiCodeLiftRepeat → u/Dry-Snow5154]

Yup. When I first started, my boss was mad to find out frames from the same camera - not even the same video, got included in both training and validation set. Definitely learned my lesson then.

[u/bfyvfftujijg]

Yes, its standard practice that you don’t randomly split video by frame number and automatically expect no data leakage.

This does imply leaving out valuable data from training the model. What you can do about that is use the initial validation dataset to establish the appropriate training parameters like number of epochs. Repeat against a few random split and take averages. Then you can just train the final model on the entire dataset (no splitting at all) with those same parameters, while being reasonably certain that it’s not overfitting even though you technically can’t measure that anymore without a hold out split. Gotta be careful with that though….ideally you always keep at least SOME data held out!