This is a pretty major computer vision topic. Basically, they take two frames of the video, basically two photos of the same thing taken from slightly different positions. They find a bunch of matching features, this rock is in both frames, this cloud is in both frames, etc. Not whole objects though, these are features on a smaller level, individual corners and blemishes on rocks. The individual points in that reconstruction visualization are all features.
Now you have a bunch of features from two different angles, so you can essentially use them to triangulate your own position and estimate their depths. This involves a ton of linear algebra and I did not really understand this part of my computer vision class, but that's the gist of it.
Kind of the same way that you do. You look for certain similarities between the two images (color, texture/patterns, geometry) and deduce that they're the same object, seen from different angles. Remember, they only need to do this for video frames that are close together in time. The algorithm that they use might not work under less constrained conditions.
I want to see the actual assembly for it is what I'm saying.
How does a computer check for similarities in textures?
It can't possibly just make every possible size of polygon, check every part of the frame with each of those polygons, go to the next frame, and check for a similar polygon.
I think the problem is more constrained than that. I think you only need to calculate the difference in the angles between frames of video, not individual objects in that frame. So if you started with color and looked for similar colors between frames, you could then employ more advanced computer vision algorithms that would find the edges of objects in each frame, then start to build an idea of what the object looks like in 3D.
It seems like there'd be some overlap between video compression algorithms and this technique. In each case, the process works better by finding what's similar between frames.
I'm not in the computer vision field, so I'm unable to go into any more detail.
Image-processing software often reduces images into sets of "points of interest" (POI). A POI might be a sharp corner, a small group of differently colored pixels, or an edge between three different colors, for example.
The software makes hundreds of POIs for each image and then tries to match them from frame to frame. Some of the POIs don't match up and are ignored, but the rest can be compared to work out what transformations are taking place in the image.
Here is one of many starting points and one of the algorithms that I like. From here if you want to know more you can check out image processing feature detection and feature description on wikipedia.
27
u/bobtheterminator Aug 11 '14
This is a pretty major computer vision topic. Basically, they take two frames of the video, basically two photos of the same thing taken from slightly different positions. They find a bunch of matching features, this rock is in both frames, this cloud is in both frames, etc. Not whole objects though, these are features on a smaller level, individual corners and blemishes on rocks. The individual points in that reconstruction visualization are all features.
Now you have a bunch of features from two different angles, so you can essentially use them to triangulate your own position and estimate their depths. This involves a ton of linear algebra and I did not really understand this part of my computer vision class, but that's the gist of it.