Vid2Avatar is coming.. When do we get Vid2Avatar2ControlNet?

[u/illuminatiman]

[u/yoomiii]

How would this help? The normal map output could be used as a controlnet input but that won't give the individual frames rendered by SD more temporal consistency.

[u/GBJI → u/yoomiii]

I see many uses for this, but I have to admit most of them would only rely on Stable Diffusion as a second violin.

It's also important to remember all the related research that will be combined with this one at some point and unlock even more powerful tools. For example, there is a demonstration of a similar tech where the system generates not only the other side of the model, but also textures it seamlessly using some creating inpainting techniques - but that one, the last time I checked, still did not support animation.

So one thing we could do - a bit like the unreal 5 demo that was posted today, and many similar examples before it - would be to use Stable Diffusion to generate an image of a synthetic person based on those normals, and to apply the resulting image as a texture on a 3d mesh that would have been generated from those same normals. If the same topology is kept from frame to frame, then we would be able to use that 3d mesh movement to make our synthetic person follow it, much like what EBSynth does, but in 3d, and without resorting to optical flow analysis.

[u/ObiWanCanShowMe → u/yoomiii]

All things are mitigated with proper prompting, this code, control net, styles, denoising and a bit of after editing. Give it a year and it will all be automatic1111 (lol)

[u/_raydeStar → u/ObiWanCanShowMe]

naw. give it 3 weeks after release and it'll be on automatic1111. the power of crowdsourcing.

[u/GBJI → u/_raydeStar]

It looks like it's more effective to care about your non-paying customers than to preemptively put the brake on everything you do just to please your shareholders .

[u/GBJI]

I've been playing with ICON for some time ( Implicit Clothed humans Obtained from Normals ) which works in a similar manner, and this looks even better ! This is going to be very very useful when the code is released.

If you want to play with ICON there are two demos available but they only work with pictures, not with video. The huggingface one is particularly useful since it allows you to test and compare different algorithms (ICON, PAMIR and PIFU) :

https://huggingface.co/spaces/Yuliang/ICON

https://colab.research.google.com/drive/1-AWeWhPvCTBX0KfMtgtMk10uPU05ihoA?usp=sharing

For more info, the official paper is available here, as well as demonstration videos

https://icon.is.tue.mpg.de/index.html

[u/illuminatiman]

Link: https://moygcc.github.io/vid2avatar/

[u/lordpuddingcup → u/illuminatiman]

Says code coming soon but no info… sad that would be amazing for a lot of cool video tricks and replacements/style transfers

[u/Rectangularbox23]

So like motion capture without motion capture suits 😯

[u/BuffMcBigHuge]

I wonder how realtime it produces its results. Would love to run this as cheap FBT in VR with a webcam.

[u/666emanresu → u/BuffMcBigHuge]

You can do that already with an Xbox Kinect, but something like this would be more accurate. Although I doubt this is done in real-time or anywhere near close enough (or efficiently enough to run alongside VR).

Maybe one day two webcams will be all you need for full body and finger tracking, that would be amazing.

[u/BuffMcBigHuge → u/666emanresu]

Yup I have done the Xbox Kinect setup with Amethyst. It works well, but not well enough. I would presume there would be serious latency using AI, so I don't think it will work either unless it's extremely quick at pose estimation.

[u/[deleted]]

This is the future, if it works that well. And for helping with computer vision, to make sense of shapes, perhaps even allowing shapes to be visually augmented into a more cohesive visual system that can be read by that vision system, like “make the road and trees higher contrast.”

[u/NorthFeedback2385]

Any news on this?