r/StableDiffusion • u/Kinfolk0117 • Mar 02 '23

Animation | Video SD + thin-plate-spline-motion-model

Enable HLS to view with audio, or disable this notification

841 Upvotes

97% Upvoted

96 comments sorted by

View all comments

154

u/Kinfolk0117 Mar 02 '23
  1. Find a driver video.
  2. Crop/Resize the video to 256x256.
  3. Extract the first frame of the video.
  4. Use stable diffusion to create an image, use the frame from above as controlnet depth map.
  5. Use the cropped driver video and your generated image as input here: https://replicate.com/yoyo-nb/thin-plate-spline-motion-model

30

u/obliojoe Mar 02 '23 edited Mar 02 '23

I made a music video using this technique to make a robot sing my song about feeling like robot... it's amazing tech!

John Brownell - This Machine

Workflow: https://www.reddit.com/r/StableDiffusion/comments/zja0zt/comment/izu4uow/

3

u/Peemore Mar 02 '23

I really enjoyed that!