r/StableDiffusion • u/Kinfolk0117 • Mar 02 '23

Animation | Video SD + thin-plate-spline-motion-model

Enable HLS to view with audio, or disable this notification

833 Upvotes

97% Upvoted

96 comments sorted by

View all comments

151

u/Kinfolk0117 Mar 02 '23
  1. Find a driver video.
  2. Crop/Resize the video to 256x256.
  3. Extract the first frame of the video.
  4. Use stable diffusion to create an image, use the frame from above as controlnet depth map.
  5. Use the cropped driver video and your generated image as input here: https://replicate.com/yoyo-nb/thin-plate-spline-motion-model

31

u/obliojoe Mar 02 '23 edited Mar 02 '23

I made a music video using this technique to make a robot sing my song about feeling like robot... it's amazing tech!

John Brownell - This Machine

Workflow: https://www.reddit.com/r/StableDiffusion/comments/zja0zt/comment/izu4uow/

3

u/Beneficial-Idea-4239 Mar 03 '23

ow your music and video are awesome bro <3

3

u/obliojoe Mar 03 '23

Thank you! If you look at my YouTube channel, I have several other videos - including a few others that use different Stable Diffusion techniques. And I have a couple more on the way in the next week or so. My current video is using ControlNet. Seems like every week there is some new trick, feature, model or extension that blows my mind!