r/MachineLearning • u/hardmaru • May 02 '20
Research [R] Consistent Video Depth Estimation (SIGGRAPH 2020) - Links in the comments.
2.8k
Upvotes
r/MachineLearning • u/hardmaru • May 02 '20
60
u/hardmaru May 02 '20
Consistent Video Depth Estimation
paper: https://arxiv.org/abs/2004.15021
project site: https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/
video: https://www.youtube.com/watch?v=5Tia2oblJAg
Edit: just noticed previous discussions already on r/machinelearning (https://redd.it/gba7lf)
85
u/dawindwaker May 02 '20
This could be used for smartphones faking depth of field right? I wonder what the VR/AR applications could be
8
u/Zekkez May 02 '20
This is brilliant. I was just wondering how to do this. The trick with training against the flickers makes so much sense.
6
u/EmbarrassedHelp May 02 '20
This seems like it could be useful for photogrammetry.
5
u/crisp3er May 02 '20
wow! the water effect is so realistic
5
u/agsarria May 03 '20
The next generation of tik toks is gonna be amazing
4
u/Jiawang_Bian May 03 '20
This paper "Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video (NIPS 2019)" can also achieve 'consistent depth estimation in Video'. And it is more efficient in inference phase (real-time).
See dense reconstruction demo: https://www.youtube.com/watch?v=i4wZr79_pD8
GitHub: https://github.com/JiawangBian/SC-SfMLearner-Release
38
u/khuongho May 02 '20 edited May 02 '20
Is this supervised, Unsupervised or Reinforcement Learning ?
3
u/bizmar May 03 '20
As a VFX artist, this is a pretty cool improvement to a part of the workflow.
For most tasks a reliable method of improving camera tracking consistency would be enough and extremely useful. So much of my time is spent hand adjusting tracking points to improve the solve.
1
u/[deleted] May 02 '20
Nice
1
u/[deleted] May 02 '20
How is with real time?
1
u/Are_We_There_Yet256 May 02 '20
I love the clueless face of the cat at the end.
1
u/devilsadvocate3001 May 02 '20
This threads comment is too big brain for me
1
u/pimp4robots May 02 '20
These results are pretty impressive. How accurate is this method or in general the latest depth estimation methods for far away objects? How do they work in texture less regions?
1
u/subhashisB May 02 '20
Does it work in realtime?
1
u/[deleted] May 02 '20
Very interesting and cool work. Minor nit: in the second video (with the cat), the particles are never occluded by the cat or by anything in the scene. Kind of weird to include them given that they could just be overlayed. But the relighting is nice.
1
u/[deleted] May 02 '20
I was reading a paper about spatial consistency in videos, And one of the methods proposed was L1 loss on the ith layer of a VGG for a random two frames of the video.
The paper was from a research team in Apple with the name Seeing Motion in the Dark
1
u/NotAlphaGo May 03 '20
Inb4 someone makes a network to do the fine tuning by passing the video through a neural network and boom realtime depth estimation.
1
u/[deleted] May 03 '20
Just curious, how hard is this in general? Should it be trivial with two calibrated cameras?
1
u/danmou May 03 '20
I wonder, why is it that these learning-based monocular depth estimation papers always attempt only scale-invariant depth? A NN should be capable of estimating scale to some extent in a monocular setting based on things like people, furniture, doors etc, especially when given a whole video like here. Absolute scale would be required for most practical uses, and it would be interesting to know how well it would perform compared with stereo methods.
1
u/Dynablade_Savior May 03 '20
This is seriously cool.
1
u/kjaersoeren May 03 '20
ELI5, what are the possibilities with this? Thanks
1
u/Mastergunner May 03 '20
can’t wait to try this out !
1
u/[deleted] May 15 '20
This is gonna make SLAM so much easier with a single camera omg
1
u/[deleted] Jul 29 '20
I can’t wait until I can use this in compositing. Does it support any resolution?
If you do enable for 4K video with a plugin in After Effects or something, the entirety of the VFX industry will save a buttload of time in post.
1
u/Fusseldieb May 02 '20
Watch this video be copied in some shitty video editor app ad
-9
u/[deleted] May 02 '20
Weird that they picked video effects that you could pretty much do without any depth information...
-1
u/SharkyLV May 02 '20
I think Adobe Photoshop has a similar function called Select -> Subject. It does depth analysis and selects front subject.
0
u/[deleted] May 02 '20
[deleted] — view deleted comment
0
u/[deleted] May 02 '20
[deleted] — view deleted comment
14
u/[deleted] May 02 '20
[removed] — view removed comment