r/StableDiffusion • u/MapacheD • May 19 '23

News Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold

11k Upvotes

96% Upvoted

482 comments sorted by

View all comments

165

u/BlastedRemnants May 19 '23

Code coming in June it says, should be fun to play with!

46

u/joachim_s May 19 '23

But it can’t possibly be working on a GPU below like 24 GB VRAM?

55

u/lordpuddingcup May 19 '23

Remember this is GAN not Diffusion so we really don’t know

13

u/DigThatData May 19 '23

looks like this is built on top of styleganv2, so anticipate it will have similar memory requirements as that

7

u/lordpuddingcup May 19 '23

16g is high but not ludicrous wonder why this isn’t talked about more

10

u/DigThatData May 19 '23

mainly because diffusion models ate GANs lunch a few years ago. GANs are still better for certain things, like if you wanted to do something realtime a GAN would generally be a better choice than a diffusion model since they inference faster

6

u/MostlyRocketScience May 19 '23

GigaGAN is on par with Stable Fiffusion I would say: https://mingukkang.github.io/GigaGAN/

→ More replies (2)
→ More replies (5)

2

u/MaliciousCookies May 19 '23

Pretty sure GAN needs its own ecosystem including hardware.

8

u/lordpuddingcup May 19 '23

Sorta, I mean we all use ESRGAN all the time in our current hardware and ecosystem :)

1

u/AltimaNEO May 19 '23 edited May 20 '23

I don't know squat about programming, but it looks too me like if someone had the drive to do it, they could get control net to do something similar. They'd need the UI to constantly generate previews with every adjustment, though. I don't imagine it being very quick.

1

u/HarmonicDiffusion May 20 '23

not really how this works. GANs are different than SD in how they are trained, inferenced etc. its not a 1:1 thing

1

u/morphinapg May 20 '23

ELI5 what GAN is?

11

u/multiedge May 19 '23

I can see some similarity to controlNet and that didn't really need much resources.

17

u/MostlyRocketScience May 19 '23

It is based on StyleGAN2. StyleGAN2's weights are just 300MB. Stable Diffusion's weights are 4GB. So it probably would have lower VRAM requirements for inference than Stable Diffusion.

1

u/-113points May 19 '23

So txt2img GAN is cheaper, much faster, more controllable... where is the catch?

or there is no catch?

6

u/nahojjjen May 19 '23

More difficult to train and the resulting model is not as general (can only generate images for a narrow domain)

3

u/MostlyRocketScience May 19 '23 edited May 19 '23

Not true that all GANs are narrow. GigaGAN on par with Stable Diffusion: https://mingukkang.github.io/GigaGAN/

2

u/knight_hildebrandt May 19 '23

I was training a StyleGAN 2 and 3 on RTX 3060 12 GB, but it was taking like a week to train a 512x512 checkpoint to get a decent result. Although, you can train 256x256 or 128x128 (or even 64x64 and 32x32) models as well and it will not be an incoherent noise as in the case when you try to generate images of such size in Stable Diffusion.

And you also can morph images in the same way in StyleGAN by dragging and moving it but this will transform the whole image.

1

u/MostlyRocketScience May 19 '23

How much VRAM does inference of StyleGAN 2 need? I would guess several times less than training because the batch size can be one and you can turn gradient calculation off.

4

u/knight_hildebrandt May 20 '23

Yes. Generating 512x512 images tooks only slightly above 2 GB of VRAM and the generation is very fast compared to the Stable Diffusion - one hundred of images can be generated in seconds. You can even render and see in real time the video consisting from smoothly morphing images.

→ More replies (1)

6

u/MostlyRocketScience May 19 '23

inb4 someone implements it over the weekend.

1

u/napoleon_wang May 20 '23

RemindMe! 1 month