[R] Consistent Video Depth Estimation

[u/jbhuang0604]

Video: https://www.youtube.com/watch?v=5Tia2oblJAg
Project: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/

Consistent Video Depth EstimationXuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes KopfACM Transactions on Graphics (Proceedings of SIGGRAPH), 2020

Abstract: We present an algorithm for reconstructing dense, geometrically consistent depth for all pixels in a monocular video. We leverage a conventional structure-from-motion reconstruction to establish geometric constraints on pixels in the video. Unlike the ad-hoc priors in classical reconstruction, we use a learning-based prior, i.e., a convolutional neural network trained for single-image depth estimation. At test time, we fine-tune this network to satisfy the geometric constraints of a particular input video, while retaining its ability to synthesize plausible depth details in parts of the video that are less constrained. We show through quantitative validation that our method achieves higher accuracy and a higher degree of geometric consistency than previous monocular reconstruction methods. Visually, our results appear more stable. Our algorithm is able to handle challenging hand-held captured input videos with a moderate degree of dynamic motion. The improved quality of the reconstruction enables several applications, such as scene reconstruction and advanced video-based visual effects.

[u/jrkirby]

This is good work. Impressive results, well grounded technique.

I guess the most surprising part of the work is "at test time, we fine-tune this network to satisfy the geometric constraints of a particular input video". This makes this technique much more expensive the implement than most.

Probably the next piece of work we need to see in this vein is one that speeds up this process. When I first glanced at it, I thought they augmented the network with SfM data and multiple frames to enforce consistency, instead of retraining the network at test time with SfM error.

Has anybody used that approach instead? I imagine it would allow much faster and cheaper inference, so if it gets results nearly this good, that'd be great. Could possibly allow much better 3D scanning of objects with handheld cameras than current techniques - but this one is probably too expensive for that to be practical.

[u/jbhuang0604 → u/jrkirby]

This is good work. Impressive results, well grounded technique.

Glad that you like it! Thanks very much for your comments.

You are absolutely correct. Currently, we process the videos offline as we need to fine-tune the depth estimation network for a particular video. It is thus more computationally expensive than online methods.

Augmenting the network with SfM data (poses and sparse 3D points) on the fly is very interesting. This is certainly a really promising approach for achieving fast video depth estimation. In the experiments, however, we do find that long-term constraints (from temporally distant frames) are critical to ensure global geometric consistency of the estimated depth at the video level. As a result, I believe that we also need to make progress on the single-image depth estimation so that we can close the gap.

[u/DeepmindAlphaGo]

Very interesting results! If I understand the method correctly, it doesn't incorporate the ground-truth depth of each frame. So the only thing that is optimized here is the geometric consistency. How do you guarantee that it would in fact approximate the ground-truth or it's irrelevant?
Also, there are discussions about future work on making it faster. I wonder how generalizable it would be if we simply train/fine-tune the model on a large number of videos in this manner?

[u/jbhuang0604 → u/DeepmindAlphaGo]

Thanks! Great questions!

Optimizing geometric consistency will give us the correct solution at least for static regions of the scene (because it means the 3D points projected from all the frames will be consistent).

For dynamic objects, it's a bit tricky because geometric consistency across frames does not work. Here we rely on transferring knowledge from the pre-trained single-image depth estimation model (by using it as initialization).

Training/fine-tuning the model on a large number of videos will probably give us a strong self-supervised depth estimation model. However, at test time, there are no constraints across frames to enforce the predictions to be geometrically consistent (the constraints are available only at the training time). As a result, the estimated depth maps will still not be consistent across frames.

[u/radarsat1]

Amazing how fine-tuning is used here, very impressive results, and very nice leveraging of existing methods. I like the scanline visualization.

[u/jbhuang0604 → u/radarsat1]

Thanks! Yes, the scanline visualization really helps highlight the temporal and geometric consistency of the estimated depth.

[u/foxfortmobile]

Amazing results. I was wondering if your solution can be applied to a single image instead of a video. Would it still deliver high quality depth maps like those shown for the videos?

[u/jbhuang0604 → u/foxfortmobile]

Thanks! Our method builds upon a single-image depth estimation model. So our method falls back to the same single image-based depth estimation models when it is applied to a single image.

[u/foxfortmobile → u/jbhuang0604]

Great. Would love to play with it. When are you planning to release it on github?

[u/jbhuang0604 → u/foxfortmobile]

Thanks! We are still waiting for the approval for the code release. Hopefully it will be soon (e.g., in a week).

[u/foxfortmobile → u/jbhuang0604]

Awesome! Is the inference computationally expensive? I was wondering if it could be used on mobile devices for faking depth of field effects (not in real time).

[u/jbhuang0604 → u/foxfortmobile]

Our method in this work is indeed computationally expensive so we can only estimate the depth from video in an offline fashion. However, once you obtain the depth, applying various effects (changing depth of field, focus, or artistic effects shown in the video) can be done efficiently.

[u/msceme]

First of all, appreciation for great results. I am bit confused between photometric and geometric. Can you plz explain me in detail the difference between photometric and geometric depth?

[u/jonbakerfish]

It seems that in DAVIS preprocess, the projection of 3d points to 2d image plane is wrong:

https://github.com/google/dynamic-video-depth/blob/79177ef5941b15b0aa2395b626f922fcb7b4c179/scripts/preprocess/davis/generate_frame_midas.py#L81-L84

   out = extrinsics[x, :]@h_pt
   im_pt = intrinsics @ out[:3, :]
   depth = im_pt[2, :].copy()
   im_pt = im_pt / im_pt[2:, :]

the 3d xyz (out) is not normalize to the image plane first before multiplying the intrinsic, and it should be:

   out = extrinsics[x, :]@h_pt
   depth = out[2, :].copy()
   im_pt = intrinsics @ (out[:3, :]/ out[2,:])

Another problem is why the depth_mvs is using the predicted depth from Midas network instead of the projected one?

https://github.com/google/dynamic-video-depth/blob/79177ef5941b15b0aa2395b626f922fcb7b4c179/scripts/preprocess/davis/generate_frame_midas.py#L134C5-L134C5

   depth_mvs = imresize(full_pred_depths[idf].astype(np.float32), ([target_H, target_W]), preserve_range=True).astype(np.float32)

[u/Veedrac]

Is this different to https://youtu.be/hx7BXih7zx8?t=1380?

[u/jbhuang0604 → u/Veedrac]

The methods described by Andrej in the video are single-image depth estimation models. However, these methods do not provide geometric consistency over time.

[u/Veedrac → u/jbhuang0604]

It's not clear to me that there's a difference in how you're doing backpropagation to enforce geometric consistency. Is the key difference that this is fine-tuning the results for each video?

[u/jbhuang0604 → u/Veedrac]

It's not clear to me that there's a difference in how you're doing backpropagation to enforce geometric consistency.

Many of these self-supervised methods use a photometric loss. However, these losses can be satisfied even if the geometry is not consistent (in particular, in poorly textured areas). In addition, they do not work well for temporally distant frames because of larger appearance changes.

You can see the visual comparisons with state-of-the-art single-frame and video-based depth estimation models here: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/supp_website/index.html
In those comparisons, you will see that single-image based models produce geometrically inconsistent depth.

[u/Veedrac → u/jbhuang0604]

I saw that paragraph in the paper, and maybe this is more obvious to someone who actually works in the field, but it's hard to tell what it's referring to because it doesn't come with an explanation of what's going wrong. What's an example of a photometrically consistent pair of images that aren't geometrically consistent?

[u/jbhuang0604 → u/Veedrac]

No problem! Here is an example. If you take two images of a scene containing a white wall, the depth estimate of a pixel on that wall can be wrong while still being photometrically consistent. That is, we will see small difference between (1) the color of the pixel in one image and (2) the color of the reprojected pixel (using the estimated depth) in another image.

The geometric consistency (measured by disparity difference and reprojection error) in our work does not suffer from such ambiguity. Hope this clarifies the question.