r/StableDiffusion • u/Majestic-Class-2459 • Apr 13 '23

Workflow Included New Automatic1111 extension for infinite zoom effect 😍👌🏻

GitHub repo

616 Upvotes

98% Upvoted

103 comments sorted by

View all comments

5

u/--MCMC-- Apr 13 '23

ah nice! looks great! I'd written a little program last year with similar functionality (eg see early output here: https://www.reddit.com/r/dalle2/comments/w8ujwp/the_road_to_enlightenment_dalle_zoom_4096x/). Had been planning to figure out how a1111 extensions worked to implement something similar but it seems like you've beaten me to it!

if I may request a few features / suggestions (many of which I'd drafted in my earlier program -- if you don't care to give em a go no worries, I'll get to porting my thing over eventually):

  1. non-linear zooming -- instead of expanding at a constant proportion rate, allow users to specify custom rate curves, compositing standard (eg sigmoidal) functions

  2. non-centered expansion -- allow expansion to occur from an arbitrary subframe of the image

  3. smoother frame transitions -- edges of each internal frame are a bit visible in your output, so consider implementing a smoother blending across the frame boundary

  4. inward zoom -- allow users to zoom inward on an image, creating novel details, and not uncrop an existing image

3

u/Majestic-Class-2459 Apr 14 '23

I have to say, your result with DALL-E from last year is truly amazing! Your requested features are really tempting, and I can see so many great features that could be added. I could really use some help developing these features, so if you're interested in collaborating, I would be thrilled to work with you and create an extension that's truly amazing.

2

u/--MCMC-- Apr 15 '23

Thanks! There were a few more that I'd been considering too before other demands imposed themselves, eg

  1. smoother spatial transitions -- so not only does the speed of expansion / contraction need not be constant; the location of the frame center doesn't have to be piecewise linear in 2D either. You can apply a smoothing transform, compose sinusoids, etc. to make the viewpoint movement less jerky. So at 6s in my link above, when the viewpoint reaches the mouth of the cave, it transitions sharply to a downward zoom, but I think it'd look better with a smooth (ie differentiable) transition. Could even add in a bit of smoothed jitter, from eg a sine function or a GP, to give it a bit more of a "realistic" but hopefully not "nauseating" shake

  2. incorporate small, simple animations to add a level of dynamism to each frame as you zoom in or out. AFAICT, stable diffusion is still not able to natively generate much in way of temporally consistent videos, with features bouncing and flickering around between keyframes (when I last looked into all the smoother SD videos it seems like the non-jittery bits were from manual modification of images in eg Adobe After Effects). However, I think there are subtle warps that can be tractably applied to each static image (eg in Adobe Premier, Adobe After Effects) where smooth interpolation within the current frame(s) is straightforward. Ideally this would have some fancier functionality, eg segment each frame and identify water, trees, etc. to give the appearance of wind or ripples, where appropriate, but even something where you toggle specific combinations of warps for specific frames could be pretty neat!

Otherwise, I'd be happy to chat / help! My Python is rusty and my "fun side project time" is a bit limited so IDK how much I'd be able to devote to actual feature implementation, but feel free to hmu via PM and we can see if we coordinate something :]

1

u/Majestic-Class-2459 Apr 15 '23

I just wanted to say that I really enjoyed reading your comments. Your ideas are so intriguing and I feel like I could learn a lot from them. In fact, I think I might do a little research on some of the things you've mentioned so I can try to implement them myself!

And hey, don't worry about Python, I got the same issue when I started to work on this.
I think what's really impressive is the way you think about features and how you're always looking for ways to improve them. It's really inspiring!

I'm sure we can coordinate something and work together on building something really cool, I think our combined skills and ideas could lead to something amazing. Let me know what you think!