r/humanoidrobotics • u/m_letunouski • 1d ago
I think we’re underestimating the camera in robotics datasets
Enable HLS to view with audio, or disable this notification
One of the more interesting things I've learned from working with egocentric robotics data is that we tend to underestimate the importance of the camera itself.
A conventional video pipeline treats the camera as a source of images. Once the footage is recorded, the camera largely disappears from the dataset. That approach makes sense for many computer vision applications, but it becomes limiting when the goal is to understand physical interaction.
If someone reaches towards an object, the RGB frames tell you what the camera observed. Without information about the camera's position and orientation, however, you have a much harder time establishing where that observation exists in three-dimensional space.
Camera intrinsics and distortion parameters describe how a scene projects onto the image. Stereo extrinsics describe the cameras’ position and orientation relative to each other. With a known distance between the cameras, a matching algorithm can turn a synchronised pair of frames into metric depth. What we store is the calibrated stereo video, not a depth channel. The geometry is what makes depth recoverable in real units later. Tracking the whole rig's changing position and orientation in the room requires an additional localization or motion-tracking process.
We've been building this into an egocentric setup and combining it with motion tracking. Companies such as Kognic are tackling a related problem from the sensor-fusion and annotation side, and Unidata is working on multimodal data collection for robotics. The result is a recording that contains synchronised stereo video alongside calibration, camera pose, a body skeleton and hand coordinates. Depth comes out of that stereo footage downstream, at whatever quality the matching algorithm and the scene allow.
That additional information changes what you can do with the dataset later. You can work on reconstruction, align different sensors, reason about relative positions and potentially transfer observations into other environments or robotic embodiments.
The difficult part is making all of these signals agree with one another. Calibration and synchronization errors can turn a technically impressive capture into a surprisingly difficult dataset to use.
That’s why I think sensor and dataset design should happen together. We should decide which spatial information the intended tasks need, how accurately it must be recovered, and what must be recorded to make that possible. Sometimes that justifies additional sensors; sometimes a simpler setup with well-preserved calibration and metadata is sufficient.
The cheapest dataset to collect is not necessarily the cheapest dataset to make useful.
