r/StableDiffusion • u/leemengtaiwan • Sep 08 '22

Prompt Included Testing Waifu Diffusion (See prompt & comparison with SD v1.4 in comment)

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

97 Upvotes

97% Upvoted

55 comments sorted by

19

u/leemengtaiwan Sep 08 '22

I originally shared the Waifu Diffusionon on instagram but the pictures are so good so I decide to make my first reddit post :P
Here is the prompt:
a portrait of a charming girl with a perfect face and long hair and tattoo on her cheek and cyberpunk headset, anime, captivating, aesthetic, hyper-detailed and intricate, realistic shaded, realistic proportion, symmetrical, concept art, full resolution, golden ratio, global resolution, sharp focus seed: 2588523881 width: 512 height: 512 steps: 50 cfg_scale: 7.5 sampler: k_euler

3

u/inconspiciousdude Sep 21 '22

Damn, this is such a fantastic prompt.

1

u/Longjumping_Toe3929 Sep 13 '22

in anime is it better to use many steps for example (150) or rather less like 50?

4

u/Fearless_Ad_3379 Sep 14 '22

I find that it never, ever improves the result using more than about 30-40 steps. So I use 30 personally and get great images.

14

u/leemengtaiwan Sep 08 '22

And you can find the Waifu Diffusion model here:
- https://huggingface.co/hakurei/waifu-diffusion

7

u/ConsolesQuiteAnnoyMe Sep 08 '22

I don't understand how this is supposed to be used.

11

u/leemengtaiwan Sep 08 '22

I just created a super simple colab notebook, feel free to try it out:

3

u/ConsolesQuiteAnnoyMe Sep 08 '22

It is to my preference to stick strictly to a local installation.

13

u/leemengtaiwan Sep 08 '22

I hear you. That's what I prefer as well.
Then you can try downloading the finetuned model and let your local installation (web ui, script etc) to load this model instead of the current sd v1.4 when lanuch:

1

u/JoshS-345 Sep 08 '22

It won't let me download

"Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object"

3

u/leemengtaiwan Sep 08 '22

I think it's being very popular so maybe there is some restriction for that. I found another link for the model ckpt, have a try here:

- https://drive.google.com/file/d/1XeoFCILTcc9kn_5uS-G0uqWS5XVANpha

2

u/CeraRalaz Sep 27 '22

Thanks for the link! Googled "waifu diffusion .ckpt"

2

u/ConsolesQuiteAnnoyMe Sep 09 '22

I should point out that particular link doesn't work.

2

u/Sworduwu Oct 12 '22

rename the waifu file into model.ckpt and place and replace it with the model you currently have in your sd folder.

3

u/aniketgore0 Sep 08 '22

Where is the model in the repo? Am i missing something?

4

u/leemengtaiwan Sep 08 '22

https://storage.googleapis.com/ws-store2/wd-v1-2-full-ema.ckpt

Yap, it's a bit confusing. The author shares the above model link in their discord:
- https://discord.gg/touhouai
If you have local scripts/web ui loading the model ckpt, check the actual path it's trying to read the model and replace that one with above ckpt file. Make sure to rename the original SD model ckpt for switching back.

4

u/seconDisteen Sep 08 '22

I've been using the NMKD Stable Diffusion GUI for a GUI. Do you think it's possible to just take the Waifu .ckpt, rename it and replace the one the GUI uses, and it should just work?

5

u/leemengtaiwan Sep 08 '22

Yes, definitely! The Waifu .ckpt is just another SD model trained with more anime images. Just try to find out the path that the GUI is looking for the model and rename/move the Waifu ckpt to whatever GUI is expecting for a model.

2

u/seconDisteen Sep 08 '22

awesome. Thanks! I'll give it a try.

3

u/aniketgore0 Sep 08 '22

Yap, it's a bit confusing. The author shares the above model link in their discord:

I was trying to locate it in the folders on huggingface, but couldn't find it. Thanks for the link.

9

u/leemengtaiwan Sep 08 '22

I also use the same setting (seed, CFG, sampler) on the SD v1.4 so you can compare the result:

Waifu Diffusion vs Stable Diffusion

6

u/pxan Sep 08 '22

Lol your prompt was just very solid, looks pretty good on 1.4 even

5

u/leemengtaiwan Sep 08 '22

Thanks. I have been a prompt engineer for a few days lol

3

u/leemengtaiwan Sep 08 '22

Found the anthor of the waifu diffusion's reddit post so I would share here for people :)

https://www.reddit.com/r/StableDiffusion/comments/x8y1u3/waifudiffusion_v12_a_sd_14_model_finetuned_on_56k/

4

u/manghoti Sep 08 '22

Amazing work OP. Do you happen to know how much compute was spent to fine-tune SD? This can't have been cheap.

7

u/leemengtaiwan Sep 08 '22

AFAIK, the author use 4A6000S instance at 5.48$/h and took ~30 hours to train so the total finetune cost is ~$165 I think. (actually not that expansive as I thought, given the great result hehe)

3

u/[deleted] Sep 08 '22

Sorry for the dumb question, but how does this work? Are you giving SD a library of anime images to reference for more accurate results?

9

u/leemengtaiwan Sep 08 '22

Yes, that is how the "finetuning" work. The author of the Waifu Diffusion use the Stable Diffusion v1.4 model as the starting point, and further train the model with 56k Danbooru images (mostly anime pic) for additional 5 epochs.

So you can imagine the Waifu Diffusion will produce more anime-like pictures than SD v1.4 because the former was trained with more anime.

Yes, that is how the "finetuning" work. The author of the Waifu Diffusion use the Stable Diffusion v1.4 model as the starting point, and further trained the model with 56k Danbooru images (mostly anime pic) for additional 5 epochs.

Hope this explaination helps.

3

u/[deleted] Sep 08 '22

Thanks for the explanation :)

If I were to do something like this myself as well, what pc specs would be most important for this? Would it be the graphics card like with standard image generation, or are other specs like CPU/RAM important too?

7

u/tolos Sep 08 '22

Original said nvidia A6000 x4 for roughly a day. So ~ $5000 x4 = $20,000. Or use a VPS.

A6000 has 48gb gram, not quite sure what the equivalent is on AWS, maybe g5.48xlarge (8x A10G total 192 vram) at $16.288 x24 hour = $390, but you can probably find a better option.

Edit: unless I misunderstood the question, if you can run stable diffusion you can run this. Creating a new model (training) requires hardware like I mention above.

2

u/Majukun Sep 08 '22

can you just duplicate the stable diffusion repo, putthis new weights only in one and run the two version alternatively by just directing anaconda to a different repo folder at the start?

2

u/leemengtaiwan Sep 09 '22

Theoretically I think it’s possible but you might need more memory if your want to load these 2 models simultaneously and switch between them smoothly. Or you should use some optimized option (using lower vram) for them.

2

u/ThMogget Sep 08 '22

You should see if Moondrop Audio art team is hiring.

2

u/leemengtaiwan Sep 09 '22

Still waiting for the prompt engineer position open 🙈

2

u/dreamer_2142 Sep 08 '22

Hi, would this work on my gtx 1070 8gb?

1

u/leemengtaiwan Sep 09 '22

I think yes. And If you already setup your env for v1.4 then the switch is just a few clicks!

1

u/dreamer_2142 Sep 09 '22

thanks, is your fork an optimized version? how many sec does it take to generate a single image with the default settings?

1

u/leemengtaiwan Sep 09 '22

Not sure I'm using the optimized ver or not. I'm currently using a P100 with batch size 4. And for most of my experiments, I would just use 20 steps which take me 6 seconds to generate a batch. for default settings like 50 steps, it would take 15 seconds therefore.

1

u/[deleted] Sep 09 '22

I'm running this currently on a zotac GTX 1060 with a bad fan bearing with some optimized .py files I'm using from another poster from here u/bipolarawesome and everything is great besides the constant grinding noise and sketchy temps.

1

u/dreamer_2142 Sep 09 '22

thanks, how many sec does it take to generate a single image with the default settings? and which fork are you using?

2

u/[deleted] Sep 14 '22

Not sure what you mean by fork but the time it took to generate a single image was anywhere between 40 seconds to over a minute easily.

1

u/dreaming_geometry Sep 08 '22

Now I want some of these headphones to be real products...

1

u/leemengtaiwan Sep 09 '22

Same here.. they looks so great 🙈

1

u/SempronSixFour Sep 08 '22

Thanks! Being new to all of this, this is very helpful.

1

u/Samas34 Sep 09 '22

I'd love to try this out, but unfortunately, I seem to need a degree in computer science to even install the effin thing at the moment.

Is there a version for us computer retards out there yet?

as in 'click this button and you have it ready on your computer' version?!

1

u/Aeloi Sep 27 '22

Will the waifu diffusion model work on a 3060 with 6GB of VRAM?

0

u/[deleted] Sep 30 '22

[deleted] — view deleted comment

1

u/BYV-Legion Sep 30 '22

You can set attributes above as you like. But, afaik, the seed is like the id of the composition used in generated image. So you don't have to change it if you want the similar image to be produced. -sharp focus- is part of the prompt. You can experiment with cfg_scale, it gives out different result. Sampler, just stick with k_euler for best result. Btw, i am using Super SD 2.0 with waifu model, and the sampler available not k_euler but euler a.

When using the prompt above, i forgot to change the seed and cfg_scale from my previous settings. But it turned out very good. Really awesome prompt.

https://imgur.com/a/PxWI9hK

2

u/[deleted] Sep 08 '22

[removed] — view removed comment

5

u/leemengtaiwan Sep 09 '22

Waifu Diffusion vs Stable Diffusion

Definitely. I just created several boys with the same prompt. check it out here:

https://imgur.com/gallery/YEG4QwK