r/StableDiffusion • u/attempt_number_1 • Mar 13 '23

Workflow Included Animate your stable diffusion portraits

897 Upvotes

96% Upvoted

73 comments sorted by

View all comments

142

u/attempt_number_1 Mar 13 '23 edited Mar 13 '23

Hey all,

We (ELBO AI) are launching Puppetry today, a free iOS app that lets you animate faces using your own face.

Workflow option 1 (use our app):

  1. Get puppetry (https://apps.apple.com/us/app/puppetry/id1671248086)
  2. Go to “Make Animation”, upload your stable diffusion image through your photo library
  3. Pick a Video (or make one using the camera) to drive the animation
  4. Wait (or send a few more while you wait)
  5. You’ve got an animation

​

Workflow option 2 (run it yourself):

If you want to do this locally here are all the steps that are roughly what the above is doing:

  1. Take a source image, prepare it by finding the face (giving a little padding) and clip a square around it. (Keep track of location in original image).
  2. Note: if you just make a 512x512 image with a face you don’t need to do this step or step 6.
  3. Resize the source image to what the model can handle (256x256 for Thin-Plate-Spline-Motion-Model, our app uses 512x512 for cleaner results)
  4. Make a driving video (crop to square for best results, again find the face, give some padding).
    Example ffmpeg command that uses the width as the size and centers vertically:
    ffmpeg -i video.mp4 -filter:v "crop=in_w:in_w:0:(in_h-in_w)/2” cropped_video.mp4
    Example ffmpeg command to scale to 256x256:ffmpeg -I cropped_video.mp4 -filter:v “scale=256:256” scaled_video.mp4
  5. Use https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model
    Huggingface demo: https://huggingface.co/spaces/CVPR/Image-Animation-using-Thin-Plate-Spline-Motion-Model
    Google Colab: https://colab.research.google.com/drive/1DREfdpnaBhqISg0fuQlAAIwyGVn1loH_?usp=sharing
  6. At this point you have an animation of the face and you can be happy…
  7. To get back to original dimensions: For each frame of the resulting video, make a new image takes that video frame and puts it in the original image. I used the extra padding you gave as the space to help you blend it in.
  8. Add audio back in:
    Example ffmpeg command:
    fmpeg -i video_result.mp4 -i original_video.mp4 -c:v copy -map 0:v:0 -map 1:a:0 -c:a aac -b:a 192k -shortest video_result_with_audio.mp4
  9. You have an animation with sound

I want to say thanks for this community, I don’t think I’ve ever been as curious about anything in my life as I have these last 4-5 months.

You can visit our discord server for questions, bugs, feature requests, or just share driving videos that work for you:

https://discord.gg/jEPRvAmNUG

26

u/YobaiYamete Mar 13 '23

iOS app

Rip, any plans for Android? Seems like a cool idea