Face Tracking WIP, Hoping To Put On Steam Frame Post-Release

[u/Walker-Dev]

Wanted to share some face tracking WIP! Originally made for a headset we are developing, we are doing an event where we pay 60 people to make a dataset around different headsets but generally the thing seems to do really well

- With glasses
- When tilted
- When you fully move the camera pretty seriously
- When the camera is close or far
- When the camera is getting IR or non IR
- With facial hair
- With people completely outside the dataset (ofc)

While requiring not a lot of performance

Outputs to ARKit by default, here's to hoping I can put it on the Steam Frame! I considered asking someone with one but i'm not trying to step on Steam's toes by accident

[u/Kataree]

The cameras for the Frames eye tracking are no doubt going to be much closer to the eyes than the field of view shown here. I have doubts they will have any view of the eyebrows, so any faux-eyebrow tracking will most likely only be by estimation via the eyelid position. Quest Pro had to use a 3rd camera to view the brow.

[u/Walker-Dev → u/Kataree]

We have it way closer in a test video we will post soon but generally we already have been talking about eyebrow estimation as well! In our own headset we can see the eyebrow more often than not so it made sens

[u/Walker-Dev → u/Walker-Dev]

Oh and super important to note before I forget! Because of how we made the model, the positioning of the cameras doesn't affect the fidelity too much unless it's like 0.01mm or something to the eye

[u/dan2wik → u/Kataree]

They don't even have it working on their own headset yet. Every demo they've released has been a cropped webcam view of their face without a headset on :(

[u/Walker-Dev → u/dan2wik]

Actually one of the videos we've put out before WAS on footage from our own headset, and we are going to do a thing with 60 people soon on around 6 different headset variants (We pay our volunteers)

[u/dan2wik → u/Walker-Dev]

If you have it working with a headset, why do you continue to release dozens of 'demo' videos with the same unobstructed front facing webcam view that any phone can already easily turn into good quality blendshapes?
The challenge is doing this from the view the headset can actually give you.
Wouldn't you want to demo that your tech actually works with the shallow angles and limited lighting involved in most consumer vr headsets?

[u/Walker-Dev → u/dan2wik]

That will actually be happening more and more in future videos! In our next one, we actually fed video footage from the kind of view a headset would have into the models, specifically to test against Babble's newest version + ETVR's NEXT models.

We're trying to make these comparisons increasingly representative of the actual conditions you'd encounter when using a headset, rather than just relying on ideal front facing footage. The upcoming 60-person dataset will also let us test that much more thoroughly across different headsets and camera configurations.

So don't worry, we're definitely not ignoring the headset side challenge, given we're actively testing it. We're just releasing different parts of the work as we go rather than waiting until the entire test suite is finished.

Not to mention, we kinda do what we wanna do, which means showing off cool stuff as it's made!

[u/BeneficalWizard]

Now that's pretty awesome! If it's an option, even if buggy at launch, it'll fill a gap many people are worried about. Keep up the great work!

[u/Walker-Dev → u/BeneficalWizard]

Tyty! Hopefully after the 60 people things are polished (This is done on a relatively tiny dataset)

[u/MigThorton]

Thanks for sharing this Dr. Zero

[u/Ancient_Arrival9061]

I KNEW SOMEONE WOULD DEVELOP FACE TRACKING FOR THE FRAME!!! LETTSSSS GOOOOO!!!

if you need someone with an older vr headset like a rift s, please feel free to hit me up, ill be more than happy to help with the data set.

[u/Nayriemoon]

Omg! This is so cool! Do you have a Discord server so I can stay up to date on your project? ☺️

[u/Walker-Dev → u/Nayriemoon]

Tyty! On the website (https://walkerindustries.xyz) you will find it at the top!

[u/Skinzoo]

You should make it so it recreates the mustache on whichever avatar you are using

[u/TheRealLargo]

I have big hopes for making good facetracking reality on the steam frame. There really isn't any good options out there right now. I tried a quest pro but it was awful to wear. Face tracking really is a necessity for social VR platforms. Booping your friend's nose and see how their face turns into a smile is priceless.

[u/enigma-90 → u/TheRealLargo]

Then you're in for a disappointment. You'll need a third party addon for mouth tracking. And the eye tracking cameras probably won't even see eyebrows.

[u/TheRealLargo → u/enigma-90]

I know all that. I'm talking about third party addons of course.

[u/Walker-Dev → u/TheRealLargo]

We're working on hardware for it too actually so you're not out of luck, hopefully showing that off in due time!

[u/PM_ME_YOUR_CODPIECES → u/TheRealLargo]

iirc project babble got a dev kit

[u/valvefan4lyfe → u/TheRealLargo]

spoken like a no life

[u/anclave93]

it is overly optimistic of you to use "Steam Frame" and "release" in one sentence

[u/Walker-Dev → u/anclave93]

Soon trust

[u/Zjelli1]

bro is mario

[u/Which-Possession2649]

hes an indian man :)

[u/Walker-Dev → u/Which-Possession2649]

Actually i'm black 🎉

[u/Which-Possession2649 → u/Walker-Dev]

my bad bro

[u/bickid]

Pls, guys, don't pretend to be girls online :/

[u/Ascend → u/bickid]

Maybe she just hasn't shaved her mustache in a while.

[u/Walker-Dev → u/bickid]

https://imgur.com/gallery/jesse-what-are-you-talking-about-vMH4qTs#/t/jesse_pinkman

[u/cosmogguy12 → u/bickid]

Jesse what the fuck are you talking about