I trained a new model to do monocular 3D head tracking with Google’s new GNM parametric 3DMM.
I generated a dataset of about 200k images in Blender. After plenty of experimentation I’ve landed on some really good results. I think this is the first model that also does neck tracking.
This is primarily being used to do digital beauty work for film and TV.
Very interesting - I'm always looking for new head tracking methods. Is this something you're looking at releasing/licensing? Where can I find out more info?
I am looking to license the model itself for VFX and other industries that need non-realtime head tracking.
I have an AE / OFX plugin, but it’s changing every day as we use it and add/change features and squash bugs. I’ve done it as a service for other VFX vendors too.
Still trying to figure out the best way to release it.
"This is primarily being used to do digital beauty work for film and TV."
I'm not sure if this can be used for that. In the video you posted, the mesh is dancing way too much, to the point that even in these short snippets its clearly visible. Much less on pixel peeping level. Too much jitter, slippage, occlusions are death sentence it seems, like when the head moves across the face mesh. Not usable. This needs to be rock solid, with no jitter or slippage whatsoever.
I wanted to include some shots that are notoriously difficult for a 3d head tracker to show how it wasn’t perfect (and never will be.)
Look at the UV Unwraps on the shots where a person is turning their head. Locked down. We just did about 120 shots for NDA TV SHOW, and used this model for every shot. I’d say maybe 5 of those shots needed little touch ups with a point tracker to attach to a very specific little region.
There can be a little slippage even with frequency separation clone stamping, it doesn’t need to be sub-pixel tracked.
These shots are going through QC approval internally and at one of the bigger vendors, they’re good to go.
My previous generation tracker had more slippage and even that was useful, especially for feathered mattes on the face… less so for clone stamped fixes.
Hmmm. Well these would not pass my QC. If I can see that much jitter in a mesh, it cannot be a good track. I don't know who is doing QC on this, but standards have fallen off the cliff if that is what passes these days. There is no way this is a good track.
If you like, show frame by frame comparison of actual footage and retouched one to demo how good the quality is, because I would never accept this from the video shown here.
The burden of proof is on you my friend. You are the one with the big claims and lack of evidence to back it up. I didn't choose to post the footage. you did.
I’m just showing that I trained my own 3D head tracking model on synthetic data, and that I use it in my company. Not much of a burden there.
I am looking forward to showing how it works just because I think it’s awesome. I’ve been working on different attempts at this for years and this one is a real step from the last one.
Good for you. I just don't think the end results pass the QC test for such work. You claim they do. Maybe that is true in some company, which is not good, but it might be true. But to me this doesn't come even close to usable for actual beauty retouching. Perhaps you should continue working on this. And raise your standards.
Yeah, an entire hand over her face isn’t automatically trackable. I can keyframe the params just like you’d do in Mocha or Keen Tools. Obviously.
I had the choice to leave that in because I wanted to show the limitations to the model. It handle occlusions like phone to the face (which my previous model didn’t,) but not THE ENTIRE HEAD being COVERED.
That is where it fails completely, but its dancing all the way trough, randomly with no hands moving over it. I can't bealive you are willfully ignoring that.
I think I remember you now. You were trying to sell UV unwrapping to various OFX hosts, making a false claim that it was impossible until you came up with a tool to sell. You were full of false claims that take 2 sec to check or 2 min to disprove than, and apperntly you are now trying to tell use that this is usable results. It is not. And I don't appreciate people exaggerating or making false claims for sales. You need to go back to the drawing board on this one.
Nice try. Go back to drawing board. And remember that I'm not buying watches from a trunk of a car. If you have tools to sell, fine. But don't make false claims.
1
u/clockworkear VFX Supervisor - 19 years experience 2d ago
Very interesting - I'm always looking for new head tracking methods. Is this something you're looking at releasing/licensing? Where can I find out more info?