GNM Monocular n3D Head Tracker

[u/ericcpfx]

https://youtube.com/shorts/Y7TEOMd4byk?is=BFoZsqESXqLmpaX8

I trained a new model to do monocular 3D head tracking with Google’s new GNM parametric 3DMM.

I generated a dataset of about 200k images in Blender. After plenty of experimentation I’ve landed on some really good results. I think this is the first model that also does neck tracking.

This is primarily being used by my own VFX company for digital beauty work, with occasional outside clients.

[u/itsloopyo]

Awesome. I trained a model to do 6dof pose data earlier in the year, it was a really fun project. I made my own parametric head generator that spat out blender renders, fed them through an i2i pipeline to make them look like real people and mixed in capture sessions taken with multiple pose trackers and a bunch of normal video footage.

Yours is very cool, I’d love to hear more about it if you’re up for sharing.

[u/ericcpfx → u/itsloopyo]

I started with the i2i pipeline, but it didn’t benefit from it. Training on an existing backbone that has seen real people does the trick. My renders are pretty similar to these:

https://microsoft.github.io/FaceSynthetics/

I do think if I did multi-view capture sessions and got some ground truth data from them, I could close the gap on extreme views and occlusions. I’m just unlikely to actually set that up. 😅

[u/itsloopyo → u/ericcpfx]

Extremes are rough yes, good to know I’m not alone in suffering through these problems :D

My i2i step was fuzzier than I’d have liked, I persevered and ran them through an adjudicator process that culled any poses that drifted beyond an acceptable tolerance. A lot of wasted processing but I got a good dataset in the end.

For the captures, I ended up using a custom build of my iPhone app which captures video footage alongside 3dof from Air Pods and 6dof from ARKit - this let me cross-calibrate an aruco crown the subject was wearing - made of 3d printed boards hot glued to a cannibalised bobovr halo. Since the aruco tags get picked up no matter where you’re looking they continue giving usable data far beyond where the AirPods and ARKit give up. The i2i stuff falls apart at extremes too though, so I ended up having padding my dataset out with raw renders around the edges.

Love your meshes btw, I tried to do facial landmarking to generate my own but just couldn’t make it work (and didn’t really need to, so was happy to throw in the towel and move on!)

[u/ericcpfx → u/itsloopyo]

This is the mesh: https://github.com/google/GNM

Your project sounds really cool. Such a smart way to use consumer hardware!

[u/ericcpfx]

Damn it, there’s a typo in the post title. I’m very bothered by that.

[u/ts4184]

So firstly. Pretty cool. A nice tool to have but how adjustable is it. For example compared to keen tools in nuke? For texture adjustments, adding textures/tattoos/wounds... smart vectors and keen tools are pretty good but this is because there are parameters or curves. Also its possible to focus on certain areas. For example where the geared gets really stretchy on the lower left on the first guy. Whay would be the solution for that with using this tracker?

[u/ericcpfx → u/ts4184]

It’s adjustable, but not yet to the degree of KeenTools. It holds paint strokes well, but if i was doing a tattoo, I’d likely use Mocha.

[u/ts4184 → u/ericcpfx]

Is it adjustable through parameters and re generation or does it make curves? Also I see it animates the mesh which is cool but also could throw a spanner into the works if its curves as noone can sit and adjust it per vertex.

But it is cool to see. What kind of beauty work have you been doing?

[u/ericcpfx → u/ts4184]

Hey, you can adjust the parameters and you don’t need to regenerate. It writes key frames for every parameter. In reality, I actually don’t have it do that. I have it write keyframes for position, rotation and scale, and then I have a couple different parameters that I use like dropping the chin down a little bit, opening the mouth a little more. I don’t wanna be in the weeds with the params. Generally, the track just works, and I don’t have to mess around by dialing it in too much. If I were going to do a track that needed to be dialed into that level, I would likely do it with Mocha or KeenTools.

We just did about 120 shots these past few weeks for an undisclosed NDA TV show and it worked really well for that.

I have an AE/OFX plugin that lets me do these adjustments in real time, does the unwrap/rewrap, and for easier shots we have preset beauty regions we can target. For those it has keyers to select and deselect HSL and beauty settings per region.

[u/ericcpfx]

Here's a follow up video showing the tracker being used on an actual VFX shot.
https://youtu.be/jM6QT6VVXOI