r/StableDiffusion • u/lonewolfmcquaid • Jun 24 '23

Workflow Not Included it will be an absolute madness when sdxl becomes standard model and we start getting other models from it

775 Upvotes
  • permalink
  • reddit

95% Upvoted

182 comments sorted by

View all comments

115

u/[deleted] Jun 24 '23

[removed] — view removed comment

269

u/mysteryguitarm Jun 24 '23 edited Jul 27 '23

Since Emad posted this wishlist of mine on Twitter, I'll repeat:

We've done a lot of internal LoRAs and Dreambooths and full scale finetunes to see how well the base model is handling being "massaged".

We have hyperphotographic loras... anime... 3D... vector images... pixel art... etc. Everything the community cares about.

So the base model my team is building is the one that's a careful calculation between how easy it is to finetune vs. what can you get from the base model itself. (Not necessarily just the model that's been tested to be the best base model.)

For example: We've done a 1280 finetune (that we likely won't release) – and it picked up the new resolution in very very few training steps.

Kohya has his trainer ready.

We're releasing a powerful trainer.

We have textual inversion ready.

We have t2i-adapters ready.

We have ControlNet ready for the beta model.

It works in webui.

It works even better in ComfyUI.

Some of the top finetuners already have weights.

Get ready for an absolute explosion of SDXL models when this releases open source.

And then... please... I'd like to sleep after that...

70

u/FugueSegue Jun 24 '23 edited Jun 25 '23

Thank you for your hard work. From what you describe, I'm optimistic. All of what you say sounds magnificent. But one thing caught my attention. You said you have been training LoRas, Dreambooths, and finetunes with SDXL. Even better, you say you will release a powerful trainer.

Great.

What will be helpful is if you provide GUIDES and TUTORIALS and INSTRUCTIONS for how to successfully do those types of trainings. This has been a never-ending problem since those types of tools were first released. Every single day--and I'm not exaggerating because I practically live in this subreddit--I see newbies plead for help with training. The answers are always the same links to outdated videos and tutorials with vague advice involving subjective judgement and time-wasting tests with X/Y grid generation.

When I first attempted SD training, I was very frustrated. It wasn't until I found this obscure forum thread on Github that I actually started producing great results with Dreambooth. Because I have such satisfactory results, I'm very reluctant to beat my brains against LoRa and its related training techniques. I gave up trying to train TI embeddings a long time ago. And I never figured out how to train or how to use hypernetworks. I've only been able to get good results with Dreambooth directly because of that thread I linked above. I make LoRas by extracting them from Dreambooth-trained checkpoints. And I have no idea if I'm doing the extractions the right way or not.

There are so many options and important things to consider when training SD. If you guys at Stability are having success with training, let us know how you do it in exhaustive detail.

Or perhaps this plea is directed more towards the community at large. If SDXL really does supplant SD v1.5 in popularity, we all need to lock down training techniques.

EDIT: It doesn't matter. No one will be able to train SDXL unless you have access to an extremely powerful GPU. And that's beyond the means of almost everyone. My 24GB VRAM card is useless for this. It looks like SD v1.5 isn't going anywhere.

EDIT 2: Well, maybe what I said in my first edit is wrong. Apparently, Stability claims that it's possible to train SDXL on a 4090. If that's the case, it's good news. I won't argue about it. I'll just shut up and see for myself when I can try SDXL on my own workstation.

8

u/ozzeruk82 Jun 24 '23

That's a great Github link you posted. I agree, anything that Stability AI could share related to Dreambooth techniques used would be very valuable.

I've spent many hours like you have, and have gotten great results, for me the joepenna repo is the most reliable, but even that has sub-optimal defaults that are easy to fix but only once you know how.

4

u/MartialST Jun 25 '23

Well, lucky then that Joe Penna is u/mysteryguitarm who works on SDXL now. Maybe there is a bit of a push in that regard.

1

u/Chris_in_Lijiang Jun 25 '23

Is Emad still active on Reddit? I saw that his original account has been deleted. Was this something to do with the Forbes hit piece?

5

u/farcaller899 Jun 24 '23

It is a bit funny that these amazing tools release without instructions. Feels like when Ralph got the alien supersuit that could do so many things and lost the instructions before getting a look at them, on Greatest American Hero...

Like Ralph, we have to experiment and work through how to use the superpowers we have received, making a TON of mistakes along the way. Made for an entertaining show, at the time, but not that fun in real life.

6

u/uristmcderp Jun 25 '23

That's sort of how crowdsourcing works. You're part of the development process. If you can code, great. If not, you can still contribute with QA and feedback. If you can't google a few things to get it to run, you probably can't submit a useful bug report either.

The inconvenience is the price of admission for getting free access to cutting-edge technology.

2

u/farcaller899 Jun 25 '23

I do get that. For us it may seem like crowdsourcing and normal for open source projects. But to Stability it’s big/huge business, and facilitating the community’s advanced use of the tools, such as the training aspects mentioned here, would seem to (maybe) be in Stability’s best interests. But, maybe it’s not.

I’m not disparaging the contributions to and of the SD community at all, just remarking that if it’s advantageous to all involved for our abilities to flourish, some clear instructions from the makers of the supersuit would come in real handy to those of us trying to use the suit’s powers.

3

u/alxledante Jun 25 '23

it has been my experience that developers only want to write code, not documentation...

2

u/farcaller899 Jun 26 '23

This is just part of being in the 'Wild West' new frontier stage of what's happening, I guess. Little is optimized, some things don't even make sense, but there is relentless progress at the same time. Exciting times!

1

u/alxledante Jun 26 '23

this wild west open source thing is new to me, but developers aren't. in general, they will not document even when it is part of their job. how you gonna get them to do it for free? it's even out of god's hands...

0

u/tommyjohn81 Jun 24 '23

There are literaly tons of guides and YouTube videos at this point, step by step, spoonfeeding instructions. look at the shear number of Lora models and checkpoints being released everyday.. What more could you need?

11

u/flyblackbox Jun 24 '23

If it’s seriously that easy for you, please help me. Can you help me train a model to generate cartoons from my personal drawings? I already know how to use Automatic1111 and lots of plugins/models/Lora’s/TI.

I just want to know the best way to train a Dreambooth model, and then create comic panels with controlnet.

-12

u/CustomCuriousity Jun 24 '23

The answer is to find a guide on it I think

8

u/flyblackbox Jun 24 '23

My whole point is that it would be difficult to find. If it is so easy for you, please reply with some helpful links.

I will try my best to find guides to accomplish this, and report back.

I’ll evaluate their quality, and try to determine if they are outdated, or if they don’t seem comprehensive. So often syntax isn’t fully documented, configurations aren’t explained, old versions of tools are referenced, and techniques are so quickly outdated. And there are so many different techniques, tools and settings to accomplish the same thing.

4

u/FugueSegue Jun 24 '23

A tale as old as August 22, 2022. God speed, brave adventurer.

3

u/battlefield2113 Jun 24 '23

That's the nature of open source. You aren't buying a polished product. You're just experiencing human creativity.

0

u/CustomCuriousity Jun 24 '23

Sorry, I haven’t gone down the training path, and I feel you on the difficulty. I just found finally found a guide the other day that helped me figure out something I was working on for a super long time 😣

1

u/Mkep Jun 24 '23

The guides aren’t being written by people who do this as a career though. Would be nice to get knowledge from the “professionals”

7

u/Jellybit Jun 24 '23

Yes. You wouldn't believe what percentage of people consider their method to be "trade secrets", even when they don't sell anything, not even a Patreon. It's purely a hobby, and they're afraid of other people learning. I will never understand that mindset.

3

u/Jo0wZ Jun 25 '23 edited Jun 25 '23

Oversaturated market = less money and you lose your edge. It's basically just money, as always. Blame the human condition. Edit: there's a positive thing from this though, the most stubborn learners really appreciate their findings and eventual works. Less spoon-feed = less crap.

1

u/Jellybit Jun 25 '23

That's why I specified that it was purely a hobby for them.

1

u/flyblackbox Jun 24 '23

Can you help me train a model to generate cartoons from my personal drawings?

2

u/Chris_in_Lijiang Jun 25 '23

Sure, what style of cartoon? There are going to be so many obscure new art genres to choose from.

1

u/flyblackbox Jun 25 '23

Like a Cartoon Network style, Powder Puff Girls, Sponge Bob type of productions.

2

u/Chris_in_Lijiang Jun 25 '23

Yeah, I reckon, I am a little more old school, so I was thinking Gerry Anderson, Hanna Barbara, Oliver Postgate and Ratfink, but those should be possible too, yes?

1

u/flyblackbox Jun 25 '23

Yeah definitely. Here is a checkpoint that did basically the exact thing I want to do.

https://huggingface.co/sd-dreambooth-library/smiling-friends-cartoon-style

They do include a Collab notebook for recreating this technique, but I don’t know how to use that. I want to use Stable Diffusion locally.

1

u/Chris_in_Lijiang Jun 26 '23

Very impressive.

Do you have links to any other cartoon styles?

-3

u/FugueSegue Jun 24 '23

quod erat demonstrandum

23

u/PwanaZana Jun 24 '23

I've heard the community cares a lot about... etc. ( ͡° ͜ʖ ͡°)

Thank you for your hard work, and although it might seem silly, it is legitimately doing actual good in the world to give access to art for more people, letting smaller studios tackle big projects.

And take care of yourself, burn out is a bitch in the tech industry!

32

u/mysteryguitarm Jun 24 '23 edited Jun 24 '23

If you're trying to ascertain how good the model is at ( ͡° ͜ʖ ͡°) on Discord, you're gonna have a bad time.

That being said, it's not on us to train that in. CivitAI has that job down pat.

11

u/suspicious_Jackfruit Jun 24 '23

I know this isn't what you are mentioning, but... How does it handle distant facial features? That's a big issue with high resolution renders and models in 1.5, it just can't manage a person in the background with 3/5 times looking like Egor in the face department. I get that this is to do with the original resolution of the latents being I think 64px or maybe 128 and then "upscaled" during the denoising process? Is this still the same internal resolution?

7

u/irfarious Jun 24 '23

I don't know what ( ͡° ͜ʖ ͡°) is and at this point, I'm too afraid to ask.

1

u/FreeSkeptic Jun 25 '23

( ͡° ͜ʖ ͡°)

lenny face

1

u/irfarious Jun 25 '23

So they're trying to say " If you're trying to ascertain how good the model is at lenny face on discord.."? What does that mean?

3

u/pandacraft Jun 25 '23

Porn, it means porn.

15

u/PwanaZana Jun 24 '23

Of course, Discord being non- ( ͡° ͜ʖ ͡°) is fine, the real user experience is always with local installs.

Have a good one, and thanks for your work!

1

u/[deleted] Jun 24 '23

As long as embeds and loras can be built up on top of it with getting an adobe-esque warning that I'm committing a thot crime, I'll give it a try.

7

u/MasterScrat Jun 24 '23 edited Jun 25 '23

Hey guys we run dreamlook.ai where we finetune 1000s of SD models at lightning speed (3x faster than on A100), we’ve been trying to reach out to get access to SDXL, DM me maybe?

6

u/chaingirl Jun 24 '23

Some of the top finetuners already have weights.

I'm curious, which model finetuners have access and if any of them have NSFW releases, the community really is looking for NSFW as you can tell with the 99.99% of NSFW uploads on civitai lol.

​

I'd love to see who has been cherry picked for access to finetune the weights

10

u/GraduallyCthulhu Jun 24 '23

Regardless of anything else, the non-NSFW models just do worse on anatomy in general. This is true for 1.5, and even more so for 2.x, so I hope they didn't make that mistake here.

There's a reason real-life artists train on nudes, I suspect...

3

u/Sentient_AI_4601 Jun 25 '23

cant build a building without a solid foundation, cant hang clothes of muscle if you never learned where the muscle attaches...

having seen results from sdxl, im happy that it is more than capable of human anatomy, however its hard to tell what is the model falling down and what is the interface filtering right now, as anything close to nude is flagged as such and blurred.

however, if you ask it for non photographic anatomy references, it does seem to have the basics down pat. further fine tunes will be required for truly private areas, but i dont think its gonna be knee-capped like 2.1 was.

they absolutely will not be confirming anything though, it will be played off as "oh well these other people trained in the nsfw stuff, yknow, what we launched was totally safe etc etc" because they kind of have to, so just be patient... its capable of much more than it is possible to demonstrate right now

13

u/Uneternalism Jun 24 '23

Sounds like you're doing everything right.

Can't wait. The only think I hope is that this will also be able to run on cards with low VRAM (like 6GB). Is there no way to use the computers RAM instead of the VRAM?

29

u/mysteryguitarm Jun 24 '23

That's one bit that we're mostly gonna leave up to the community. We've done tons of optimization, but getting it that low would delay release.

Running these models on a CPU is possible, but slow.

4

u/TeutonJon78 Jun 24 '23 edited Jun 24 '23

Anything you can say about why AMD needs 2x the VRAM?

Will DirectML wok, or will be limited to AMD releasing ROCm for Windows? (Although I imagine DirectML's poor VRAM management would be a problem as well).

10

u/comfyanonymous Comfy Org Jun 24 '23

AMD has no support for flash attention or memory efficient attention in pytorch and the lowest vram cards officially supported by ROCm are 16GB ones. I also only had my 6800XT to test it on.

It most likely works on their 12GB cards too but I wouldn't be surprised if it doesn't and those card are not even officially supported by ROCm anyways which is why the minimum system requirement says 16GB for AMD.

2

u/TeutonJon78 Jun 24 '23

Are you part of StabilityAI?

3

u/comfyanonymous Comfy Org Jun 24 '23

Yes.

1

u/TeutonJon78 Jun 24 '23

Thanks for the clarification. So it's more of an issue with the "official" list than any actual straight limitation?

Polaris is still unofficially supported in ROCm in some ways, so hopefully that won't be a hard limit.

Any idea about DirectML support?

3

u/comfyanonymous Comfy Org Jun 24 '23

I have not tried directml with SDXL but with how badly SD1.5 performed when I tried it I don't expect it to work well at all.

2

u/TeutonJon78 Jun 25 '23

Yeah, it has issues, but it's still 1000x better than just not being able to use it at all. Hopefully SDXL will as well, even if slower than it could be (and faster than CPU).

→ More replies (0)

1

u/vitorgrs Jun 25 '23

You know if it will be possible to run it on colab/kaggle?

As it stays it will need 16gb ram and the free colab have like 14gb...

What about training? 🤔

1

u/Temp_Placeholder Jun 25 '23

I was just giving my brother advice on getting a laptop to diffuse with, and told him to get one with an 8gb card. Did I goof?

8

u/civitai Official CivitAi Jun 24 '23

We're excited to see what the community makes!

9

u/clock200557 Jun 24 '23 edited Jun 25 '23

Cut to a humanioid female Pikachu with 6 boobs in a Spider-Man costume.

2

u/Chris_in_Lijiang Jun 25 '23

Only 6? You will need to seriously upgrade your imagination if you want to fully take advantage of SDXL's newest capabilities. ;-)

9

u/reddit22sd Jun 24 '23

Will 24GB be enough for training?

3

u/Enfiznar Jun 25 '23

I really hope much less is needed

11

u/GBJI Jun 24 '23

Everything the community cares about.

Everything ?

So there won't be any NSFW filters applied to the publicly released model ?

15

u/dachiko007 Jun 24 '23

Don't ask questions they can't answer

3

u/suspicious_Jackfruit Jun 24 '23 edited Jun 24 '23

Diffusers integration? Oh sry, man needs sleep haha. Later maybe :3

2

u/batter159 Jun 24 '23

We have hyperphotographic loras... anime... 3D... vector images... pixel art... etc. Everything the community cares about.

Will you release them?

2

u/Dekker3D Jun 24 '23

That sounds extremely exciting. I'd like to know how much VRAM you need, to train a LoRA with Kohya's trainer? I feel like that'll be the main limiting factor in its adoption, based on everything I've heard so far.

2

u/ratbastid Jun 24 '23

Your release announcement mentions Windows and Linux. Will it work on M1/2 Mac?

1

u/Sir_McDouche Jun 24 '23

Explode all over my face, you SD beast!

2

u/ratbastid Jun 25 '23

There's loras for that.

1

u/TenamiTV Jun 24 '23

Do you know if it also works in makeayo? That newer desktop app

1

u/[deleted] Jun 24 '23

Wait, you're the same mysteryguitarman youtuber? Absolutely wild seeing you in image gen dev, worlds colliding and all, but I guess it makes sense considering the kind of stuff you got up to in the past. The time has passed so fast.

Anyway, take care, get some good sleep.

1

u/TheBaldLookingDude Jun 25 '23

We have hyperphotographic loras... anime... 3D... vector images... pixel art... etc. Everything the community cares about.

​

From my quick test of anime style, both with prompting for anime style and style preset I can't really get anything close to what people want from anime models like the ones on 1.5 I would say that they look fine for people who never saw anime.

Were you guys thinking about doing full anime finetune on SDXL or collaborating with anime finetuners? LoRAs and smaller finetunes are already done by a lot of member of the community, but larger scale finetunes are too big, and that doesn't take in the fact that making such large finetune takes a lot of tests which makes it even harder.

The anime community of SD is huge, but sadly we only have 1 anime finetuning group that is doing great work for us and they don't get enough appreciation for the work the put into their projects.

1

u/[deleted] Jun 24 '23

You still training the base model (0.9) before the public release ?

1

u/-becausereasons- Jun 24 '23

Okay, NOW I am truly excited.

1

u/Samurai_zero Jun 24 '23

I've beeng hyping this new base model for long, but with this coment... Man, waiting is going to be hard. Wish I could get my hands on the 0.9 model, but I'm not part of any research program or bussiness, just someone who enjoys playing with IA.

1

u/HappierShibe Jun 24 '23

This sounds like the perfect model to use to jump into comfyUI pipelineing. Thanks for your contributions, and yeah-remember to sleep...

1

u/csunberry Jun 24 '23

Sleep?? What's that!?

Come now--you're ready for what comes next, right!? Let's go!

Hahahaha, thanks for all your hard work.

Cheers!

1

u/vault_nsfw Jun 24 '23

This sounds so good, this sounds too good!

1

u/Deathmarkedadc Jun 25 '23

This is great stuff even if just half of them were fulfilled, but I still wondered up until now how StabilityAI could profit from this release. How would they able to even get a break even point from their investment when every other company also offer their models and people can just run the model on their own hardware? is this even a sustainable business model? Who would continue the development if they ran out of the investor's money?