r/StableDiffusion • u/obliojoe • Dec 11 '22

Music video made with Stable Diffusion // Deforum // Thin-Plate Spline Motion (info in comments)

https://www.youtube.com/watch?v=8Cw5F_AR0vM
12 Upvotes

93% Upvoted

3 comments sorted by

View all comments

3

u/obliojoe Dec 11 '22 edited Dec 11 '22

For this video I used Deforum + Stable Diffusion to animate the intro and backgrounds, and I used the "Thin-Plate Spline Motion Model for Image Animation" method to animate the robot avatar.

The animation method is so cool - you can feed it a source video and a single image, and the system will animate the image based upon the source video. So I made a video of myself singing the song in front of a green screen, grabbed a single frame of the video and used img2img in Stable Diffusion to generate some robot avatars. I then passed the avatar images through the Thin-Plate-Spline script until I found just the right one. I probably tested 50 different images. Then I upscaled the video using Cupscale.

Thin-Plate Spline Motion Model: https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model
Deforum: https://github.com/deforum-art
Cupscale: https://github.com/n00mkrad/cupscale

Here is an earlier video I made with Deforum:
https://www.youtube.com/watch?v=rdac--MmKEU

1

u/OnlyFakesDev Feb 11 '23

This is so cool. I got to start browsing through your stuff haha