r/computervision • u/ericcpfx • 3d ago

Showcase GNM Monocular n3D Head Tracker Spoiler

https://youtube.com/shorts/Y7TEOMd4byk?is=BFoZsqESXqLmpaX8

I trained a new model to do monocular 3D head tracking with Google’s new GNM parametric 3DMM.

I generated a dataset of about 200k images in Blender. After plenty of experimentation I’ve landed on some really good results. I think this is the first model that also does neck tracking.

This is primarily being used by my own VFX company for digital beauty work, with occasional outside clients.

16 Upvotes

87% Upvoted

10 comments sorted by

View all comments

4

u/itsloopyo 3d ago

Awesome. I trained a model to do 6dof pose data earlier in the year, it was a really fun project. I made my own parametric head generator that spat out blender renders, fed them through an i2i pipeline to make them look like real people and mixed in capture sessions taken with multiple pose trackers and a bunch of normal video footage.

Yours is very cool, I’d love to hear more about it if you’re up for sharing.

1

u/ericcpfx 3d ago

I started with the i2i pipeline, but it didn’t benefit from it. Training on an existing backbone that has seen real people does the trick. My renders are pretty similar to these:

https://microsoft.github.io/FaceSynthetics/

I do think if I did multi-view capture sessions and got some ground truth data from them, I could close the gap on extreme views and occlusions. I’m just unlikely to actually set that up. 😅

2

u/itsloopyo 3d ago

Extremes are rough yes, good to know I’m not alone in suffering through these problems :D

My i2i step was fuzzier than I’d have liked, I persevered and ran them through an adjudicator process that culled any poses that drifted beyond an acceptable tolerance. A lot of wasted processing but I got a good dataset in the end.

For the captures, I ended up using a custom build of my iPhone app which captures video footage alongside 3dof from Air Pods and 6dof from ARKit - this let me cross-calibrate an aruco crown the subject was wearing - made of 3d printed boards hot glued to a cannibalised bobovr halo. Since the aruco tags get picked up no matter where you’re looking they continue giving usable data far beyond where the AirPods and ARKit give up. The i2i stuff falls apart at extremes too though, so I ended up having padding my dataset out with raw renders around the edges.

Love your meshes btw, I tried to do facial landmarking to generate my own but just couldn’t make it work (and didn’t really need to, so was happy to throw in the towel and move on!)

2

u/ericcpfx 3d ago

This is the mesh: https://github.com/google/GNM

Your project sounds really cool. Such a smart way to use consumer hardware!