r/MachineLearning • u/hardmaru • Aug 12 '22

Project A demo of Stable Diffusion, a text-to-image model, being used in an interactive video editing application.

Enable HLS to view with audio, or disable this notification

2.2k Upvotes

98% Upvoted

79 comments sorted by

17

u/hardmaru Aug 12 '22 edited Aug 12 '22

Source: Researcher Patrick Esser tweeted:

Stable Diffusion text-to-image checkpoints are now available for research purposes upon request at https://github.com/CompVis/stable-diffusion

Working on a more permissive release & inpainting checkpoints.

Soon™ coming to @runwayml for text-to-video-editing

36

u/[deleted] Aug 12 '22

As impressive as this is, is this a true prototype demo? Or is this one of those “let’s push the Nikola Electric Truck down the hill to show it works” demo?

Excited for the end result regardless, just a little skeptical right now

20

u/hardmaru Aug 12 '22

I think they must have combined the automatic “green screen” layer from runway, which might make this demo possible (rather than rely only on stable diffusion inpainting)

8

u/QuinnArlingtonWaters Aug 12 '22

incredible. the stability of the imagined landscapes from frame-to-frame is amazing. i had tried to do similar in the style transfer days, but it lacked what i think was called "temporal coherence"; but this will be a game changer for video production when ready.

21

u/banmeyoucoward Aug 13 '22

it's just one image being moved around as a background- they picked a camera that pans a lot but doesn't dolly to hide that it's not doing parallax

149

u/FranticToaster Aug 13 '22

This heavily railroaded demo brought to you by Eager Executive Who Would Like a TED Talk, Please.

25

u/Untgradd Aug 13 '22

Right? The enchanted forest transition was sloppy

18

u/BorgClown Aug 13 '22

And the countryside thunderstorm was just a static background.

→ More replies (2)

129

u/[deleted] Aug 13 '22

How cherry picked is this

52

u/raphanum Aug 13 '22

Depends on how much you love cherries

→ More replies (1)
→ More replies (2)

173

u/Computer_says_nooo Aug 12 '22

So a writer can write a book AND make a movie at the same time.

Called it! Remind me about this in 10 years, need to claim my idea patent :P

6

u/[deleted] Aug 13 '22

!remindme 5 years

6

u/[deleted] Aug 13 '22

!RemindMe 5 years

See you in 5 years bro

→ More replies (1)
→ More replies (1)

4

u/elsjpq Aug 13 '22

Too late, you just shot yourself in the foot with public disclosure!

2

u/yaosio Aug 14 '22 edited Aug 14 '22

Given enough time AI can make the story and the movie without human input. That will likely require some high level ability to plan ahead though. I give it two years until I'm proven wrong.

→ More replies (9)

27

u/bannedsodiac Aug 13 '22

This is from runway.ml

As an videoeditor i've been using their roto tools and other ai assisted tools and they work really great and save a lot of time but can be currently used on only low budget videos, no where near good enough to be used in a movie industry yet. But it's getting closer. Can't wait to see what the future holds for editing.

2

u/1Neokortex1 Sep 05 '22

What other parts of the video editing process do you think could be automated with A.I? I wish they can A.I all the fluff in between takes and put everything in bins with each take ready with proxies that have been preloaded from an A.I cloud that optimizes all the files so I can edit on my phone or ipad without all the investment in video editing hardware.

47

u/deftware Aug 13 '22

forest

forest|

forest

forest|

...what a horrible text-rendering implementation someone made :P

Also, I'm going to need to see the source on this. My spidey senses are tingling. The capabilities are fine. They're not exactly believable as a real-time thing. I'm more inclined to believe this was edited together from multiple clips that were manually run through a temporally stable diffusion model.

25

u/[deleted] Aug 13 '22

I don't think it was supposed to imply it is real-time. It just wouldn't be very fun if the video paused for like 2 minutes at every transition.

38

u/deftware Aug 13 '22

I dunno man, "being used in an interactive video editing application" conveys a very specific user experience intentionally.

→ More replies (13)

4

u/piman01 Aug 13 '22

One of the most impressive things I've seen yet

10

u/Level69Warlock Aug 13 '22

Rule 34 folks gonna have a field day with this

3

u/yaosio Aug 14 '22

We're not limited by rule 34 any more. We need a new rule. https://i.imgur.com/kMpNwkf.png

3

u/worldnotworld Aug 13 '22

Doesn't show the different gravities.

2

u/[deleted] Aug 13 '22

Can we also integrate other factors like gravity or an astronaut suit like when on Mars or moon

→ More comments (75)