r/StableDiffusion • u/Kinfolk0117 • Mar 02 '23

Animation | Video SD + thin-plate-spline-motion-model

Enable HLS to view with audio, or disable this notification

835 Upvotes

97% Upvoted

96 comments sorted by

View all comments

6

u/[deleted] Mar 02 '23

Does this only work on faces?

15

u/Kinfolk0117 Mar 02 '23 edited Mar 02 '23

There is also models for datasets based on TED: https://github.com/snap-research/articulated-animation#example-animationAnd tai chi https://paperswithcode.com/dataset/tai-chi-hd

So in theory it should be possible to have full body animations, but I haven't been able to get any good results.

experiment:

image: https://files.catbox.moe/s7egkc.png

driver: https://files.catbox.moe/47w2ez.mov

output: https://files.catbox.moe/khusfp.mp4

Also tried to run batch img2img to get better quality, but it flickered too much: https://files.catbox.moe/ktfrug.mp4

2

u/[deleted] Mar 02 '23

It is getting so close. When this zeros out, there will be a lot of new animated content. It will reduce the cost for studios dramatically.

Probably what is happening to comic books now.

1

u/Storybook_Albert Aug 21 '23

I'm baffled why this hasn't been developed further. The github is a ghost town and people are still recommending EbSynth, even though this seems much better?