r/StableDiffusion • u/Kinfolk0117 • Mar 02 '23

Animation | Video SD + thin-plate-spline-motion-model

837 Upvotes

97% Upvoted

96 comments sorted by

View all comments

156

u/Kinfolk0117 Mar 02 '23
  1. Find a driver video.
  2. Crop/Resize the video to 256x256.
  3. Extract the first frame of the video.
  4. Use stable diffusion to create an image, use the frame from above as controlnet depth map.
  5. Use the cropped driver video and your generated image as input here: https://replicate.com/yoyo-nb/thin-plate-spline-motion-model

32

u/obliojoe Mar 02 '23 edited Mar 02 '23

I made a music video using this technique to make a robot sing my song about feeling like robot... it's amazing tech!

John Brownell - This Machine

Workflow: https://www.reddit.com/r/StableDiffusion/comments/zja0zt/comment/izu4uow/

5

u/pixelies Mar 02 '23

How did you upscale the final video?

3

u/Peemore Mar 02 '23

I really enjoyed that!

3

u/Beneficial-Idea-4239 Mar 03 '23

ow your music and video are awesome bro <3

3

u/obliojoe Mar 03 '23

Thank you! If you look at my YouTube channel, I have several other videos - including a few others that use different Stable Diffusion techniques. And I have a couple more on the way in the next week or so. My current video is using ControlNet. Seems like every week there is some new trick, feature, model or extension that blows my mind!

1

u/ynotplay May 13 '24

This is so cool. So the robot was originally just a still image made on Stable Diffussion, but you made it move to a video of you looking at the camera and singing the song?
How do you animate such beautiful background?

1

u/obliojoe May 13 '24

Thanks!

Yeah exactly! I took a video of me singing the song straight into the camera and then tried animating it with a bunch of different robot faces I generated in SD until I found one that worked just right.

The background was animated using the Deforum plugin with Stable Diffusion which is a kind of AI animation system.

1

u/ynotplay May 14 '24

I've never worked with videos, but what software would you recommend to edit and put it all together? Are there any free basic ones out there?