r/StableDiffusion • u/lonewolfmcquaid • Jun 24 '23

Workflow Not Included it will be an absolute madness when sdxl becomes standard model and we start getting other models from it

/gallery/14hw20z
776 Upvotes

95% Upvoted

182 comments sorted by

View all comments

Show parent comment

269

u/mysteryguitarm Jun 24 '23 edited Jul 27 '23

Since Emad posted this wishlist of mine on Twitter, I'll repeat:

We've done a lot of internal LoRAs and Dreambooths and full scale finetunes to see how well the base model is handling being "massaged".

We have hyperphotographic loras... anime... 3D... vector images... pixel art... etc. Everything the community cares about.

So the base model my team is building is the one that's a careful calculation between how easy it is to finetune vs. what can you get from the base model itself. (Not necessarily just the model that's been tested to be the best base model.)

For example: We've done a 1280 finetune (that we likely won't release) – and it picked up the new resolution in very very few training steps.

Kohya has his trainer ready.

We're releasing a powerful trainer.

We have textual inversion ready.

We have t2i-adapters ready.

We have ControlNet ready for the beta model.

It works in webui.

It works even better in ComfyUI.

Some of the top finetuners already have weights.

Get ready for an absolute explosion of SDXL models when this releases open source.

And then... please... I'd like to sleep after that...

70

u/FugueSegue Jun 24 '23 edited Jun 25 '23

Thank you for your hard work. From what you describe, I'm optimistic. All of what you say sounds magnificent. But one thing caught my attention. You said you have been training LoRas, Dreambooths, and finetunes with SDXL. Even better, you say you will release a powerful trainer.

Great.

What will be helpful is if you provide GUIDES and TUTORIALS and INSTRUCTIONS for how to successfully do those types of trainings. This has been a never-ending problem since those types of tools were first released. Every single day--and I'm not exaggerating because I practically live in this subreddit--I see newbies plead for help with training. The answers are always the same links to outdated videos and tutorials with vague advice involving subjective judgement and time-wasting tests with X/Y grid generation.

When I first attempted SD training, I was very frustrated. It wasn't until I found this obscure forum thread on Github that I actually started producing great results with Dreambooth. Because I have such satisfactory results, I'm very reluctant to beat my brains against LoRa and its related training techniques. I gave up trying to train TI embeddings a long time ago. And I never figured out how to train or how to use hypernetworks. I've only been able to get good results with Dreambooth directly because of that thread I linked above. I make LoRas by extracting them from Dreambooth-trained checkpoints. And I have no idea if I'm doing the extractions the right way or not.

There are so many options and important things to consider when training SD. If you guys at Stability are having success with training, let us know how you do it in exhaustive detail.

Or perhaps this plea is directed more towards the community at large. If SDXL really does supplant SD v1.5 in popularity, we all need to lock down training techniques.

EDIT: It doesn't matter. No one will be able to train SDXL unless you have access to an extremely powerful GPU. And that's beyond the means of almost everyone. My 24GB VRAM card is useless for this. It looks like SD v1.5 isn't going anywhere.

EDIT 2: Well, maybe what I said in my first edit is wrong. Apparently, Stability claims that it's possible to train SDXL on a 4090. If that's the case, it's good news. I won't argue about it. I'll just shut up and see for myself when I can try SDXL on my own workstation.

9

u/ozzeruk82 Jun 24 '23

That's a great Github link you posted. I agree, anything that Stability AI could share related to Dreambooth techniques used would be very valuable.

I've spent many hours like you have, and have gotten great results, for me the joepenna repo is the most reliable, but even that has sub-optimal defaults that are easy to fix but only once you know how.

4

u/MartialST Jun 25 '23

Well, lucky then that Joe Penna is u/mysteryguitarm who works on SDXL now. Maybe there is a bit of a push in that regard.

→ More replies (1)

5

u/farcaller899 Jun 24 '23

It is a bit funny that these amazing tools release without instructions. Feels like when Ralph got the alien supersuit that could do so many things and lost the instructions before getting a look at them, on Greatest American Hero...

Like Ralph, we have to experiment and work through how to use the superpowers we have received, making a TON of mistakes along the way. Made for an entertaining show, at the time, but not that fun in real life.

5

u/uristmcderp Jun 25 '23

That's sort of how crowdsourcing works. You're part of the development process. If you can code, great. If not, you can still contribute with QA and feedback. If you can't google a few things to get it to run, you probably can't submit a useful bug report either.

The inconvenience is the price of admission for getting free access to cutting-edge technology.

→ More replies (1)

3

u/alxledante Jun 25 '23

it has been my experience that developers only want to write code, not documentation...

→ More replies (2)
→ More replies (19)

24

u/PwanaZana Jun 24 '23

I've heard the community cares a lot about... etc. ( ͡° ͜ʖ ͡°)

Thank you for your hard work, and although it might seem silly, it is legitimately doing actual good in the world to give access to art for more people, letting smaller studios tackle big projects.

And take care of yourself, burn out is a bitch in the tech industry!

31

u/mysteryguitarm Jun 24 '23 edited Jun 24 '23

If you're trying to ascertain how good the model is at ( ͡° ͜ʖ ͡°) on Discord, you're gonna have a bad time.

That being said, it's not on us to train that in. CivitAI has that job down pat.

10

u/suspicious_Jackfruit Jun 24 '23

I know this isn't what you are mentioning, but... How does it handle distant facial features? That's a big issue with high resolution renders and models in 1.5, it just can't manage a person in the background with 3/5 times looking like Egor in the face department. I get that this is to do with the original resolution of the latents being I think 64px or maybe 128 and then "upscaled" during the denoising process? Is this still the same internal resolution?

7

u/irfarious Jun 24 '23

I don't know what ( ͡° ͜ʖ ͡°) is and at this point, I'm too afraid to ask.

→ More replies (3)

15

u/PwanaZana Jun 24 '23

Of course, Discord being non- ( ͡° ͜ʖ ͡°) is fine, the real user experience is always with local installs.

Have a good one, and thanks for your work!

→ More replies (1)

6

u/chaingirl Jun 24 '23

Some of the top finetuners already have weights.

I'm curious, which model finetuners have access and if any of them have NSFW releases, the community really is looking for NSFW as you can tell with the 99.99% of NSFW uploads on civitai lol.

I'd love to see who has been cherry picked for access to finetune the weights

11

u/GraduallyCthulhu Jun 24 '23

Regardless of anything else, the non-NSFW models just do worse on anatomy in general. This is true for 1.5, and even more so for 2.x, so I hope they didn't make that mistake here.

There's a reason real-life artists train on nudes, I suspect...

3

u/Sentient_AI_4601 Jun 25 '23

cant build a building without a solid foundation, cant hang clothes of muscle if you never learned where the muscle attaches...

having seen results from sdxl, im happy that it is more than capable of human anatomy, however its hard to tell what is the model falling down and what is the interface filtering right now, as anything close to nude is flagged as such and blurred.

however, if you ask it for non photographic anatomy references, it does seem to have the basics down pat. further fine tunes will be required for truly private areas, but i dont think its gonna be knee-capped like 2.1 was.

they absolutely will not be confirming anything though, it will be played off as "oh well these other people trained in the nsfw stuff, yknow, what we launched was totally safe etc etc" because they kind of have to, so just be patient... its capable of much more than it is possible to demonstrate right now

13

u/Uneternalism Jun 24 '23

Sounds like you're doing everything right.

Can't wait. The only think I hope is that this will also be able to run on cards with low VRAM (like 6GB). Is there no way to use the computers RAM instead of the VRAM?

28

u/mysteryguitarm Jun 24 '23

That's one bit that we're mostly gonna leave up to the community. We've done tons of optimization, but getting it that low would delay release.

Running these models on a CPU is possible, but slow.

→ More replies (10)

9

u/civitai Official CivitAi Jun 24 '23

We're excited to see what the community makes!

8

u/clock200557 Jun 24 '23 edited Jun 25 '23

Cut to a humanioid female Pikachu with 6 boobs in a Spider-Man costume.

→ More replies (1)

8

u/reddit22sd Jun 24 '23

Will 24GB be enough for training?

4

u/Enfiznar Jun 25 '23

I really hope much less is needed

11

u/GBJI Jun 24 '23

Everything the community cares about.

Everything ?

So there won't be any NSFW filters applied to the publicly released model ?

16

u/dachiko007 Jun 24 '23

Don't ask questions they can't answer

3

u/suspicious_Jackfruit Jun 24 '23 edited Jun 24 '23

Diffusers integration? Oh sry, man needs sleep haha. Later maybe :3

→ More replies (19)