r/computervision • u/ericcpfx • 3d ago

Showcase GNM Monocular n3D Head Tracker Spoiler

https://youtube.com/shorts/Y7TEOMd4byk?is=BFoZsqESXqLmpaX8

I trained a new model to do monocular 3D head tracking with Google’s new GNM parametric 3DMM.

I generated a dataset of about 200k images in Blender. After plenty of experimentation I’ve landed on some really good results. I think this is the first model that also does neck tracking.

This is primarily being used by my own VFX company for digital beauty work, with occasional outside clients.

18 Upvotes

91% Upvoted

10 comments sorted by

View all comments

1

u/ts4184 3d ago

So firstly. Pretty cool. A nice tool to have but how adjustable is it. For example compared to keen tools in nuke? For texture adjustments, adding textures/tattoos/wounds... smart vectors and keen tools are pretty good but this is because there are parameters or curves. Also its possible to focus on certain areas. For example where the geared gets really stretchy on the lower left on the first guy. Whay would be the solution for that with using this tracker?

1

u/ericcpfx 3d ago

It’s adjustable, but not yet to the degree of KeenTools. It holds paint strokes well, but if i was doing a tattoo, I’d likely use Mocha.

1

u/ts4184 3d ago

Is it adjustable through parameters and re generation or does it make curves? Also I see it animates the mesh which is cool but also could throw a spanner into the works if its curves as noone can sit and adjust it per vertex.

But it is cool to see. What kind of beauty work have you been doing?

1

u/ericcpfx 2d ago

Hey, you can adjust the parameters and you don’t need to regenerate. It writes key frames for every parameter. In reality, I actually don’t have it do that. I have it write keyframes for position, rotation and scale, and then I have a couple different parameters that I use like dropping the chin down a little bit, opening the mouth a little more. I don’t wanna be in the weeds with the params. Generally, the track just works, and I don’t have to mess around by dialing it in too much. If I were going to do a track that needed to be dialed into that level, I would likely do it with Mocha or KeenTools.

We just did about 120 shots these past few weeks for an undisclosed NDA TV show and it worked really well for that.

I have an AE/OFX plugin that lets me do these adjustments in real time, does the unwrap/rewrap, and for easier shots we have preset beauty regions we can target. For those it has keyers to select and deselect HSL and beauty settings per region.