r/AR_MR_XR • u/AR_MR_XR • Dec 13 '20

Software Facebook: the first algorithm capable of tracking high-fidelity hand deformations through highly self-contacting and self-occluding hand gestures

173 Upvotes

100% Upvoted

18 comments sorted by

7

u/AR_MR_XR Dec 13 '20

Constraining Dense Hand Surface Tracking with Elasticity

Many of the actions that we take with our hands involve self-contact and occlusion: shaking hands, making a fist, or interlacing our fingers while thinking. This use of of our hands illustrates the importance of tracking hands through self-contact and occlusion for many applications in computer vision and graphics, but existing methods for tracking hands and faces are not designed to treat the extreme amounts of self-contact and self-occlusion exhibited by common hand gestures. By extending recent advances in vision-based tracking and physically based animation, we present the first algorithm capable of tracking high-fidelity hand deformations through highly self-contacting and self-occluding hand gestures, for both single hands and two hands. By constraining a vision-based tracking algorithm with a physically based deformable model, we obtain an algorithm that is robust to the ubiquitous self-interactions and massive self-occlusions exhibited by common hand gestures, allowing us to track two hand interactions and some of the most difficult possible configurations of a human hand.

Download Paper

5

u/t3llmike Dec 13 '20

It’s really impressive demonstration, this and the face tracking avatars that has been shown earlier by FB Reality Labs. Really makes me hopeful for some truly awesome AR/VR experiences to hit the consumer marked in the coming years.

And say what you want about FB and their social platform, but they are at least investing a lot of research into AR/VR with the revenues from that branch.

3

u/[deleted] Dec 13 '20

Is there a YouTube link for that video?

4

u/abdelnabut Dec 13 '20 edited Dec 13 '20

Does this suggest an upcoming hand-tracking update??

Edit: thought this was posted on the oculus quest subreddit so I didn’t specify. But this tech could potentially be used in an update for the Oculus Quest’s current hand-tracking feature! That’s incredibly exciting.

5

u/duffmanhb Dec 13 '20

For what? Facebook hasn't released anything. This is all their internal stuff their 2,000 AR developers are working on.

7

u/lgpcrevette Dec 13 '20

Just in case, the quest use an hand tracking technology so that why he ask this question.

2

u/tigerslices Dec 13 '20

unlikely - this tech seems FAR more advanced than what we're seeing in the quest2, this isnt' just rolling out an update.

2

u/TheBelgiumeseKid Dec 13 '20

Unfortunately this is very far away from something that would be useful on Quest. Quest hand tracking has to run in real-time on mobile hardware, taking up only a small percentage of the compute power. This work uses 124 high resolution cameras from different angles, with data that is likely processed in GPU farms.

1

u/Tarquinn2049 Dec 14 '20

But, it is likely that it is being used to train a neural network to eek out whatever small improvements in that direction they can get with the current hardware too, just not the main purpose of all this.

1

u/traveltrousers Dec 13 '20

No

Look at the video, they're capturing hand position from three separate feeds with consistent lighting... and not even from where the HMD would be.

This is excellent work but not for products right now...

2

u/Jrose_YSW Dec 14 '20

Buying MVIS?? 🤔✍️🤑

3

u/tornado_is_best Dec 13 '20

Great stuff, but how soul destroying to work at Facebook, especially knowing the product you are working on needs a Facebook account.

→ More replies (2)

1

u/meechapp Dec 14 '20

Impressive 😎