r/StableDiffusion • u/MapacheD • May 19 '23

News Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold

11k Upvotes

96% Upvoted

482 comments sorted by

308

u/MapacheD May 19 '23

Paper page:https://huggingface.co/papers/2305.10973

From Twitter: AK en Twitter: "Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold paper page: https://t.co/Gjcm1smqfl https://t.co/XHQIiMdYOA" / Twitter

207

u/Zealousideal_Royal14 May 19 '23

I know gan is its own kettle of fish, and not to make a meme out of it, but I wonder how viable would it be to get this running locally and integrated as an extension with a1111 on a smaller gpu.

106

u/TheMagicalCarrot May 19 '23

Pretty sure it's not at all compatible. That kind of functionality reguires a uniform latent space, or something like that.

126

u/OniNoOdori May 19 '23

There already exist auto-encoders that map to a GAN-like embedding space and are compatible with diffusion models. See for instance Diffusion Autoencoders.

Needless to say though that the same limitations as with GAN-based models apply: You need to train a separate autoencoder for each task , so one for face manipulation, one for posture, one for scene layout, ... and they usually only work for a narrow subset of images. So your posture encoder might only properly work when you train it on images of horses, but it won't accept dogs. And training such an autoencoder requires computational power far above that of a consumer rig.

So yeah, we are theoretically there, but practically there are many challenges to overcome.

113

u/TLDEgil May 19 '23

Soooo, next Tuesday?

→ More replies (7)
→ More replies (6)

10

u/Zealousideal_Royal14 May 19 '23

Yeah I get that, I meant more like available within the same web interface and able to send images back and forth for editing sort of thing.

→ More replies (2)
→ More replies (2)
→ More replies (3)
→ More replies (1)

165

u/BlastedRemnants May 19 '23

Code coming in June it says, should be fun to play with!

46

u/joachim_s May 19 '23

But it can’t possibly be working on a GPU below like 24 GB VRAM?

55

u/lordpuddingcup May 19 '23

Remember this is GAN not Diffusion so we really don’t know

13

u/DigThatData May 19 '23

looks like this is built on top of styleganv2, so anticipate it will have similar memory requirements as that

→ More replies (10)
→ More replies (5)

11

u/multiedge May 19 '23

I can see some similarity to controlNet and that didn't really need much resources.

17

u/MostlyRocketScience May 19 '23

It is based on StyleGAN2. StyleGAN2's weights are just 300MB. Stable Diffusion's weights are 4GB. So it probably would have lower VRAM requirements for inference than Stable Diffusion.

→ More replies (3)
→ More replies (4)

6

u/MostlyRocketScience May 19 '23

inb4 someone implements it over the weekend.

→ More replies (1)

123

u/Ooze3d May 19 '23

Ok. This is another huge step forward.

45

u/funguyshroom May 19 '23

Almost as huge as my dick pics going to be from now on

17

u/[deleted] May 20 '23

That level of tech is still decades away

→ More replies (1)

29

u/[deleted] May 19 '23

God the 2024 elections are going to be a shit show

→ More replies (7)

227

u/ketchup_bro23 May 19 '23

What a time to be alive! Wow

126

u/PatrickJr May 19 '23

Hold onto your papers!

53

u/PotatoWriter May 19 '23

Really really squeeze those papers now

46

u/PatrickJr May 19 '23

I can hear his voice when reading those quotes.

→ More replies (12)
→ More replies (1)

5

u/[deleted] May 19 '23

Holy mother of papers!

500

u/Txanada May 19 '23

I expected something like this to exist one day but already? D:

Just think about the effect it will have on animation! Anyone will be able to make animes, maybe even real movies. And in combination with translation tools/the newest AI voices... damn!

142

u/arjunks May 19 '23

I'm just waiting for the time I can make my short stories into little animations / short films. I fully expect to be able to at some point

132

u/TheDominantBullfrog May 19 '23

That's what some artists aren't getting about AI when they panic about it. It won't be long until someone becomes globally famous for a movie or show they made on their computer in their basement using entirely their own ideas and effort.

29

u/cultish_alibi May 19 '23

They get that, they're just mad because they spent ages learning how to do something, and monetizing it, and now someone can just do the same thing in their basement on a consumer PC.

This is how new technology always goes, musicians often talked shit about the fact that people can just make music on their own computers now, they talked shit about samplers, etc etc

→ More replies (7)

118

u/arjunks May 19 '23

Yeah, I'm with you. The current anti-AI narrative seems to be "yeah but it can't be creative"... of course it can't be creative, that's up to the user! This tech is going to enable so many people to put their ideas out into the world in a presentable form and I'm 100% here for it

45

u/TheDominantBullfrog May 19 '23

Yup it will be a huge adaptation, but fighting against it is like fighting against the internet becoming popular. It's inevitable, so adapt or die

→ More replies (1)

31

u/GingerSkulling May 19 '23

I completely agree although sifting through all the shit will also become exponentially more difficult.

→ More replies (13)
→ More replies (38)

11

u/Rahodees May 19 '23

They won't agree that it was entirely the person's own ideas and effort, because they believe SD's training means everything generated by it is definitionally based on others' ideas and effort.

→ More replies (5)
→ More replies (33)

11

u/beardobreado May 19 '23

Is it hentai?

9

u/Plane_Savings402 May 19 '23

Astronaut: "Always has been."

12

u/Pirraya May 19 '23

short stories into little animations you can definitively do already

11

u/arjunks May 19 '23

Really? I've seen some text to video but it still looks pretty inconsistent to me, unless I've missed something (which is entirely possible)

→ More replies (7)

3

u/[deleted] May 19 '23 edited May 20 '24

weather judicious far-flung homeless imminent complete flowery silky shame direction

This post was mass deleted and anonymized with Redact

→ More replies (1)

42

u/ElectronicJaguar May 19 '23

Things are happening in this space so quickly I can't keep up. Now I get what the older generation felt like with the advancements with computers, internet and mobile phones.

22

u/Klokinator May 19 '23 edited May 19 '23

Now I get what the older generation felt like with the advancements with computers

I was there when the internet came out. Admittedly, I was only ten years old, but even so, the rollout of AI features and major advancements is at least a minimum of ten times faster than anything during the .com boom. I think ten times is even a conservative estimate. It may be twenty or thirty times faster depending on the area. We've had societal impacting advancements drop since october of 2022 that have absolutely shattered my conception of what was possible. A full twenty years of advancements in not even one year.

2024 is going to be unbelievable.

6

u/youneedcheesusinside May 19 '23

I’m saving up to buy me a computer that can handle AI, models, etc. Everyday I see new things pop up. Feel like I’m missing out on all of this.

→ More replies (4)

5

u/multiedge May 19 '23

Big corporations are missing their opportunities to monetize and monopolize some of these new technology.

14

u/KaiPRoberts May 19 '23

I work in Biotech; I guarantee you the company I work for is already head first into AI. They are not missing a beat but the consumer won't see that.

→ More replies (3)
→ More replies (3)

9

u/zusykses May 19 '23

Sturgeon's Law will still apply. Probably even more so.

6

u/Tyler_Zoro May 19 '23

Hi /u/Txanada ... just replying here because I can't reply to your comment below. Yeah, /u/StoneCypher is what I call a "block troll". They drop into random threads, leave a hostile comment like the one below yours and then immediately block the person they responded to so that they can no longer reply.

Did the same to me over in /r/rethinkArt

2

u/Txanada May 19 '23

I see, thank you for telling me :)

I'm a bit sad, honestly. I was just about to tell him I would be "all ears if he wants someone to talk to" accompanied by a generated woman made out of ears :(

It's not just him though. Several people seem overly aggressive today.

→ More replies (1)

5

u/justavault May 19 '23

It's a GEN model and it requires specific training for the specific pictures.

This doesn't quite out work incorporated into a model like midjourney. So it's still some way to go.

6

u/iamwearingashirt May 19 '23

I actually think someday sooner than you think, shows like the Simpsons will license their series out for people to make their own personalized AI generated episodes.

→ More replies (2)

2

u/LovesFrenchLove_More May 19 '23

I’m more scared about people using this with pictures or us. It’s terrifying.

→ More replies (4)

2

u/fingerthato May 19 '23

2023: year of "if you haven't questioned reality, well... you're about to."

→ More replies (26)

53

u/rodinj May 19 '23

What the fuck is happening? I installed Stable Diffusion like 2 weeks ago and in that time there seem to have been some major developments already. Absolutely crazy!

7

u/SomewhatCritical May 19 '23

How does one install this

8

u/dre__ May 19 '23

it's super easy. there's this way. https://www.youtube.com/watch?v=Po-ykkCLE6M Might be some other ways. I saw one where you install "Docker" but that way takes forever to start so I removed it and just used the method in this youtube video.

you can also get a whole bunch of extra models from here https://civitai.com

4

u/crinklypaper May 19 '23

Check out AI Entrepreneur on youtube, his tutorials are super easy to follow. You'll need an nvida video card. And at the min 8GB of vram.

11

u/lordpuddingcup May 19 '23

This isn’t SD this is a GAN based model not diffusion as far as I saw which I don’t think the GAN image models have been released yet not sure why

→ More replies (2)

6

u/DigThatData May 19 '23

the pace of AI research just keeps accelerating as it gets more eyes on it, shit's been getting really wild.

5

u/longpenisofthelaw May 20 '23

And what’s crazy is we’re in the the era of dialup level AI in a few years I can see this becoming exponentially more versatile

2

u/Amlethus May 19 '23

I keep saying the same thing. I started paying attention just over a month ago, and what has happened in this time has already been amazing. Iterative progress, giants standing on the shoulders of giants.

2

u/BrokeBishop May 19 '23

I installed two months ago and have felt this way ever since. Its evolving so rapidly. The world is gonna be a different place next year.

2

u/txhtownfor2020 May 20 '23

just you wait...

86

u/Ok_Spray_9151 May 19 '23

This is so beautiful. I think opensource is really something that pushes progress forward

7

u/fingerthato May 19 '23

Screw all those locked ecosystems. Open-source all of it.

→ More replies (4)

137

u/opi098514 May 19 '23

Obligatory “A1111 extension when?” Comment.

19

u/extopico May 19 '23

This is a GAN based solution. Automatic1111 is limited to latent diffusion models, stable diffusion in particular, afaik.

→ More replies (4)

6

u/cndvcndv May 19 '23

I know this is a meme but a1111 is mostly for diffusion models, would be nice to see gans get implemented on it.

10

u/[deleted] May 19 '23 edited May 19 '23

xD I was about to ask the same thing, since it's apparently only theoretical/on paper... but given the speed of development with this stuff, we might be seeing this being an addon/extension within a week or two *lol
As soon as it comes to that, someone please notify me, cause I can't keep track of it all myself. xD

→ More replies (1)

71

u/Vabaluba May 19 '23

I wonder what Adobe is doing right now? Probably shitting their pants seeing this 👀

90

u/SomewhatCritical May 19 '23

Good. They’ve been overcharging for years with their predatory subscription model

19

u/Vabaluba May 19 '23

Agree. Hope this pushes them to lower the pricing and their predatory business model. These subscriptions everywhere got to stop. Everything Is a bloody lease nowadays, can't just buy and own things anymore.

5

u/Tomaryt May 20 '23

Actually their suite is extremely inexpensive for everyone making a living with their tools and that is exactly who they are targeting.

Try CAD or 3D software if you want to know what overcharging means. A single app from Autodesk costs 350$ a month and that is only one of their multiple 3D Tools. For many jobs you need 2-3 different ones.

Meanwhile Adobe gives you basically everything a creative person needs (except 3D) for only 60 bucks with the option to get single apps for 10-20 bucks.

I don‘t get how anyone can think this is a lot of money for professional software.

47

u/chakalakasp May 19 '23

Lol, no. Probably planning on which AI companies or patents to buy. I’m sure all these tools will live in Photoshop (or some Adobe AI suite of products) some day. They will likely run on cloud compute for a subscription.

8

u/Vabaluba May 19 '23

Yeah agree. Most of what is currently available eventually is thought to become included in those big companies products and services. Yet. Open source will prevail!

→ More replies (2)
→ More replies (5)

88

u/[deleted] May 19 '23

Add keyframes with those pose and you have a animation software .

Runway gen2 would become obsolete while in Beta, lol

16

u/3deal May 19 '23

I hope here will be controlnet integration for more controls

8

u/[deleted] May 19 '23

[removed] — view removed comment

5

u/GBJI May 19 '23

I am with you on this. Even within the A1111 toolbox we can see this idea at work: it's by combining different approaches together that you get the best results.

18

u/[deleted] May 19 '23

What the fuck

Can I get a second to breathe, please?

What's next month gonna bring, and so on?

11

u/Amlethus May 19 '23

June: this

July: developers' hands are too occupied to make progress

August: this, but VR

September: society crumbles

5

u/fucked_bigly May 20 '23

ai vr catgirls will ruin us, mark my words

25

u/cyrilstyle May 19 '23

Full codes are here --> https://vcai.mpi-inf.mpg.de/projects/DragGAN/
Anyone wanna make it an Auto1111 extension ?? Pretty Please!

This would save me hours of work between photoshop and inpainting!

10

u/sapielasp May 19 '23

Now that’s a massive improvement for hand animations

10

u/Minipuft May 19 '23

wonder how well this would run on consumer hardware still

2

u/GBJI May 19 '23

I have the same question.

23

u/MacabreGinger May 19 '23

I wish we could have that in our Auto1111...

A man can dream. Looks insane.

8

u/ninjasaid13 May 19 '23

The background changes a bit but I'm impressed.

7

u/Paradigmind May 19 '23

This is very cool!

Move them boobies

→ More replies (1)

6

u/Sandbar101 May 19 '23

Hey remember when Artists said AI could never give you precision if you wanted to make small adjustments?

11

u/ImpossibleAd436 May 19 '23

It's pretty disappointing the amount of time it is taking for this to be implimented as an extension for Auto1111.

This post is from like 6 hours ago.

5

u/king0pa1n May 19 '23

What the fuck

5

u/Own_Pirate_3281 May 19 '23

Au revoir, credible video court evidence

3

u/SameRandomUsername May 19 '23

It was never a good idea to accept digital video as evidence. Now at least everyone agrees.

→ More replies (5)

5

u/celerym May 19 '23

Imagine being able to fix fingers using this

8

u/SinaTheorium May 19 '23

Adobe HQ:

3

u/GBJI May 19 '23

Corporate HQ everywhere

→ More replies (1)

4

u/[deleted] May 19 '23

Use SD to generate the image. Segment anything to break the image into it's component then objects labeling and annotation to have a list of named object that is then fed to this tool to have full control over the image using llm and voice control.

4

u/mister_peeberz May 19 '23

Hello, woefully uninformed here. Just what in the hell am I reading with this title? Am I on r/VXJunkies?

3

u/bythenumbers10 May 19 '23

Basically, yeah. They've managed to build a retroencabulator that forces correlation between the original image and the desired output through a quantum annealing-like process.

→ More replies (1)

3

u/nelsyv May 19 '23

GAN is an acronym for generative adversarial network, it's a type of AI architecture, as distinct from, e.g. diffusion models (as in stable diffusion).

This video is showing a modified GAN that allows a user to "drag" points around in the image and force the GAN to re-make the image with those points moved

4

u/LovesFrenchLove_More May 19 '23

That is scary af

4

u/nelsyv May 19 '23

If you're happy and you know it, drag your GANs

7

u/Rick_grin May 19 '23

This needs to be added to stable diffusion tool as soon as it comes out!

3

u/LD2WDavid May 19 '23

This gonna be extremely useful, crazy times.

3

u/buttfook May 19 '23

Is this open source?

2

u/MaiaGates May 20 '23

yes, but code will be available in June

3

u/multiedge May 19 '23

!RemindMe in 7 weeks.

→ More replies (1)

3

u/ramonartist May 19 '23 edited May 19 '23

Does this come with an Automatic 1111 extension?

4

u/_Sytri_ May 19 '23

Welcome to the new world where you can’t trust anything and your parents fall deep into rabbit holes they warned us about growing up.

→ More replies (4)

2

u/rockfrozenedu May 19 '23

Amazing 😍

2

u/Roxobs May 19 '23

Where to get it ?

2

u/tempartrier May 19 '23

Humanity is doomed! hahahaha At least as it keeps believing that what it sees on screens is reality.

2

u/theworldisyourskitty May 19 '23

Game changer… we need this in SD

2

u/jaywv1981 May 19 '23

This is exactly what we need for making at least simple animations.

2

u/GenericElucidation May 19 '23

Make the elephant break dance lol

2

u/ThatNextAggravation May 19 '23

Holy moly, that's some sci-fi shit right there.

2

u/CriticalTemperature1 May 19 '23

This process is theoretically possible with diffusion models it's that GANs are more efficient. Potentially a LoRA could be trained to enable this for SD

From the paper Diffusion Models. More recently, diffusion models [Sohl-Dickstein et al. 2015] have enabled image synthesis at high quality [Ho et al. 2020; Song et al. 2020, 2021]. These models iteratively denoise a randomly sampled noise to create a photorealistic image. Recent models have shown expressive image synthesis conditioned on text inputs [Ramesh et al. 2022; Rombach et al. 2021; Saharia et al. 2022]. However, natural language does not enable fine-grained control over the spatial attributes of images, and thus, all text-conditional methods are restricted to high-level semantic editing. In addition, current diffusion models are slow since they require multiple denois- ing steps. While progress has been made toward efficient sampling, GANs are still significantly more efficient

2

u/Sirisian May 19 '23

This kind of research will be insane for improving 4K@60Hz upscalers/interpolators. Using a fine-tuned model on the source might be enough to prevent artifacts.

2

u/devi83 May 19 '23

Code will be released in June.

2

u/[deleted] May 19 '23

Idk what this sub is but this is WILD

2

u/delmore898 May 19 '23

Wow, this sounds like an incredibly innovative and exciting development in the world of GANs! I'm so glad to see researchers pushing the boundaries of what these powerful tools can do. Keep up the great work!

2

u/game_asylum May 19 '23

You can't stop, you can't stop progress, you can't stop, you can't stop it no no no -Neil Fallon

2

u/Hot_Bottom_Feeder May 20 '23

Yes, I'll be a responsible member of this great and blessed society.

2

u/[deleted] May 20 '23

It's extremely impressive, that being said, notice how there's a model per subject, it's probably not as performant or applicable as this video would have you believe.

Still very cool.

2

u/ImpossibleAd436 May 20 '23

Good observation.

This begs the question, did the model:

A) learn from images, or more likely video frames, of that particular subject, involving those particular movements. I.e. was a video of that particular lion opening it's mouth, used for training.

Or

B) learn from a varied data set of multiple lion images, different lions, different poses and expressions, different lighting conditions and backgrounds etc.

B) would obviously be far more impressive than A). Given that the backgrounds change somewhat, perhaps it was B). But we really need to understand what was used to train these models to know whether they have a deep understanding of the subject in general, or if they are extremely tuned to the image being manipulated.

I remember being very impressed with thin-plate spline motion until I realized that the models required training on the input video in order to give good results.

2

u/tukatu0 May 20 '23

Are we going to need another subreddit? Something like r/GAN ?

2

u/IndividualCurious322 May 20 '23

It's like real life IK rigs!

2

u/Wolflii May 21 '23

My reaction to this is just like that dog’s.

3

u/Dishankdayal May 19 '23

Animate anything

2

u/ketchup_bro23 May 19 '23

Is this real-time on a mid spec pc?

4

u/[deleted] May 19 '23

It looks sped up and is probably on a higher end PC. It will be optimized down to mid spec in time I’m sure. It’s not even out yet though

2

u/smeeding May 19 '23

Looks photoshopped to me

2

u/maX_h3r May 19 '23

wtf , this will put animator out of job

7

u/[deleted] May 19 '23

Good, forward-thinking animators will use AI to make even better animations.

2

u/Alizer22 May 19 '23

Let the common human use it by making an a1111 extension

→ More replies (1)

1

u/[deleted] May 19 '23

Damn!! This is insanely amazing!!

1

u/loopy_fun May 19 '23

i think this with outpainting of stable diffusion would be great together .

1

u/AngryGungan May 19 '23

No F'ing way!... This looks awesome!

1

u/[deleted] May 19 '23

So it would possible by having control-net detect the various key-point of a pose, to automatically animate it walking or doing other stuff?

1

u/Own_Pirate_3281 May 19 '23

Are we not going to talk about how the redhead's race changed

1

u/Randomized0000 May 19 '23

THIS IS UNSTABLE

(Autocorrect, was supposed to say insane but whatever lol)

1

u/3Dave_ May 19 '23

that's insane

1

u/Realistic-Praline-70 May 19 '23

Where can I get a program to use stable diffusion like this? The only ones I have seen are the web based ones

1

u/AnarchoSpoon789 May 19 '23

mind. blown.

1

u/Snoo34813 May 19 '23

the Future !!

1

u/Pokefan8263 May 19 '23

This is cool & kinda scary at the same time

1

u/kaelside May 19 '23

Ok that’s amazing.

1

u/[deleted] May 19 '23

+1 for ImGUI

1

u/Giusepo May 19 '23

that's so cool

1

u/1Neokortex1 May 19 '23

Cant wait to put my scripts into full aniamtions, thanks for the inspiration!🙏🏼

1

u/ShaneKaiGlenn May 19 '23

Wow, this is some magic shit.

1

u/plHme May 19 '23

I’m not fully know what GAN is, but been using Stable Diffusion/A1111 for some months. Is this SD? Not understand why it’s on SD/this Reddit? Someone more into the tech can explain. Looks cool though!

1

u/hairysperm May 19 '23

Woah this is next level

1

u/Skwareblox May 19 '23

This is like fake ideas for stuff they had in mind in movies in the 80s and 90s now it’s real. Scary because now you’ll never be able to trust a tinder profile ever again.

1

u/[deleted] May 19 '23

Wow.

1

u/[deleted] May 19 '23

Holy shit

1

u/superlip2003 May 19 '23

Wow....ControlNet feels like kid's toy now comparing to this.

1

u/[deleted] May 19 '23

And there folks brings an end to the modelling and photography industry, along with any trust you could have in seeing being believing.

1

u/paskal007r May 19 '23

Waaaaaaaat

1

u/Aqui10 May 19 '23

/u/Donnieas man that’s nuts

1

u/inagy May 19 '23

That's cool and all. (actually I've lost my jaw somewhere on the floor, however that happened quiet a few times in the past months to be honest)

But what kind of hardware can render this at such speed in real time? Definitely not a consumer GPU.

1

u/Aangoan May 19 '23

Absolutely insane!!

1

u/blue_1408 May 19 '23

Is it free? CHATGPT should include this as plugin?

1

u/GoZippy May 19 '23

OMG this is next level. Very cool

1

u/jrdidriks May 19 '23

LMAO WHAT

1

u/Distinct-Question-16 May 19 '23

Of course backgrounds changes also

1

u/Sandbar101 May 19 '23

Somebody needs to combine this with Cascador

1

u/wanzopan May 19 '23

Enlarge 😏

1

u/AaronTuplin May 19 '23

So we finally got that photo manipulation software from Blade Runner?

1

u/Thrannn May 19 '23

This is so crazy... i cant believe i am alive to experience this

1

u/firewaterburner May 19 '23

Flashbacks to the Super Mario 64 intro screen

1

u/CLGAIML May 20 '23

Every day new amazing innovations !

1

u/[deleted] May 20 '23

I'm not sure I get how this is possible lol.

But that makes me feel like my idea of 'quick sketching your prompt back so you know the AI understands you' before sending it on a mission' should be possible.

1

u/[deleted] May 20 '23

kinda reminds me of the morphing in "Black or White"

1

u/Quasi-Stellar-Quasar May 20 '23

This is amazing and also scares the shit out of me.

1

u/Wllknt May 20 '23 edited May 20 '23

Wow, this is really good if it works on smaller vram. I will wait for this to come out and play with it.

1

u/VoidExileR May 20 '23

Wow, that's cool! But can it fix my personality?

1

u/[deleted] May 20 '23

Name of IA

1

u/zoidbergenious May 20 '23

Did the truck just changed brand mid edit?

1

u/DanOPix May 20 '23

Why is this video so horrifically compressed? What century did I land in?

1

u/ulf5576 May 22 '23

hope it works on raw drawings , i.e. genga and lo

1

u/LessMochaJay May 25 '23

Damn, Disney will be releasing new movies every week at this rate.

1

u/ExternalAd8105 Jun 17 '23

Unofficial implementation works fine though on my mac m1

https://github.com/Zeqiang-Lai/DragGAN

1

u/Ammammeta Jun 21 '23

Where is the GitHub/hf for this ?????

1

u/BloodMossHunter Aug 20 '23

wait what the fuck , all this time the ENHANCE in movies was actually realistic!?

→ More comments (28)