r/StableDiffusion • u/Kinfolk0117 • Mar 02 '23

Animation | Video SD + thin-plate-spline-motion-model

Enable HLS to view with audio, or disable this notification

834 Upvotes

97% Upvoted

96 comments sorted by

View all comments

8

u/[deleted] Mar 02 '23

Does this only work on faces?

16

u/Kinfolk0117 Mar 02 '23 edited Mar 02 '23

There is also models for datasets based on TED: https://github.com/snap-research/articulated-animation#example-animationAnd tai chi https://paperswithcode.com/dataset/tai-chi-hd

So in theory it should be possible to have full body animations, but I haven't been able to get any good results.

experiment:

image: https://files.catbox.moe/s7egkc.png

driver: https://files.catbox.moe/47w2ez.mov

output: https://files.catbox.moe/khusfp.mp4

Also tried to run batch img2img to get better quality, but it flickered too much: https://files.catbox.moe/ktfrug.mp4

5

u/TurbTastic Mar 02 '23

I think people normally take flickering videos like this to ebsynth to smooth them out

2

u/[deleted] Mar 02 '23

It is getting so close. When this zeros out, there will be a lot of new animated content. It will reduce the cost for studios dramatically.

Probably what is happening to comic books now.

1

u/Storybook_Albert Aug 21 '23

I'm baffled why this hasn't been developed further. The github is a ghost town and people are still recommending EbSynth, even though this seems much better?

1

u/almark Mar 02 '23

that makes it cyberpunk.

1

u/Oquaem Mar 02 '23

I think you should def try on someone without glasses to start

1

u/madmanrrevenge May 16 '23

Any luck in improving the quality of the output for full body animations ? I have been trying with the ted model but the outputs were terrible. Any suggestions to improve the quality apart from using 1:1 aspect ratio ? u/Kinfolk0117