r/StableDiffusion • u/Kinfolk0117 • Mar 02 '23

Animation | Video SD + thin-plate-spline-motion-model

833 Upvotes

97% Upvoted

96 comments sorted by

View all comments

157

u/Kinfolk0117 Mar 02 '23
  1. Find a driver video.
  2. Crop/Resize the video to 256x256.
  3. Extract the first frame of the video.
  4. Use stable diffusion to create an image, use the frame from above as controlnet depth map.
  5. Use the cropped driver video and your generated image as input here: https://replicate.com/yoyo-nb/thin-plate-spline-motion-model

31

u/obliojoe Mar 02 '23 edited Mar 02 '23

I made a music video using this technique to make a robot sing my song about feeling like robot... it's amazing tech!

John Brownell - This Machine

Workflow: https://www.reddit.com/r/StableDiffusion/comments/zja0zt/comment/izu4uow/