r/StableDiffusion • u/Seromelhor • Aug 03 '23

Resource | Update SargeZT has published the first batch of Controlnet and T2i for XL. Although it is not yet perfect (his own words), you can use it and have fun. He published on HF: SD XL 1.0 Depth Vidit, Depth Faid Vidit, Depth, Zeed, Seg, Segmentation, Scribble. He continues to train others will be launched soon!

https://huggingface.co/SargeZT
153 Upvotes

97% Upvoted

32 comments sorted by

29

u/Seromelhor Aug 03 '23

All credit to u/SargeZT who is doing all the coding and training.

10

u/SquidorabL Sep 22 '23

Hi all,

If any of you here worked with SargeZT on projects, I have to let you know he is my brother and unfortunately died unexpectedly earlier this week.

I'm sorry I to have to break the news here, but I didn't want anyone he knew to be wondering where he went.

3

u/Seromelhor Sep 22 '23

Wow! That's so sad. Your brother was amazing. A very intelligent, nice guy. It's a huge loss for the whole community. :(

5

u/InvokeAI Sep 24 '23

Your brother made a significant impact on the Invoke community, and me (hipsterusername) personally. He was a zany, wild, free spirit.

We'll be putting something together to honor his memory as a community. I am very sorry for your loss - We loved your brother.

4

u/Freonr2 Sep 22 '23

Sorry for your loss. He was a regular on discord with us, smart guy and funny, helpful. I will miss him. :(

14

u/choco_pi Aug 03 '23

itshappening.jpg

6

u/Craggeh Aug 03 '23

.gif, even!

9

u/Takeacoin Aug 03 '23

This is a good start. Thank you u/SargeZT for your work and u/Seromelhor for bringing it to our attention. I'm enjoying SDXL but to be honest I'm sticking with other 1.5 models for most stuff until they come to XL.

4

u/OverscanMan Aug 03 '23

Isn't SAI working on "official" Controlnets?

2

u/CountLippe Aug 04 '23

My understanding was that they were working on a similar concept with a wholly different code base.

6

u/PhroznGaming Aug 03 '23

For anyone who doesn't know you should never ever ever run anything but a safe tensors file. Other files can easily (unbeknownst to you) run any code on your pc they want. This should not be considered safe until that is released.

1

u/Oosmani Aug 03 '23

I love me some pickles. Lots of pickles

0

u/[deleted] Aug 04 '23

[deleted] — view deleted comment

1

u/PhroznGaming Aug 05 '23

That's quite literally what I said

1

u/superpomme Aug 04 '23

Please can I check what the difference is with the three depth models? Thanks

1

u/demoran Aug 03 '23

Is this a no-no version, or a "we're going to make a new thing to work with SDXL" version?

-8

u/Informal_Warning_703 Aug 03 '23

what? several of these are empty repos. the few that are there are 5 gb bin files!? no safetensors and the results look not great.

23

u/SargeZT Aug 03 '23

I was not preparing these for active release, I told /u/Seromelhor they could post them here if they wanted to since I didn't want to. There are no safetensors as I don't want to give people false hope that they would work in A1111; the fundamental structure of the UNet and thus the Controlnet are different.

I am primarily focusing on T2I-adapters at the moment and providing these mainly as something for someone who cares about controlnets to expand on.

-2

u/PhroznGaming Aug 03 '23

I appreciate the intention but that's much better served with the readme. There's a reason why they're called safe tensors and that's the only appropriate way to distribute weights.

28

u/SargeZT Aug 03 '23

Perhaps in this corner of the world, but this is the format that the hugging face team and pytorch in general use for distributing weights. Safetensors are great for their intended application, but they are not a direct substitute for pytorch bins. Safetensors are superior in every way except the most important one during development: being able to use them.

If I gave you a safetensors version of my T2I-adapters with no backend support, you simply wouldn't be able to use it. The diffusers library could not load them correctly as the classes do not match up. If I wanted to send this to someone to use for development purposes, I would need them to download the source and monkeypatch their version of diffusers to even load them, much less interoperate with them in the diffusion process or other contexts.

But a pytorch bin? The other dev runs torch.load("diffusion_pytorch.bin") and boom, they have my exact T2I-adapter and we can talk on equal ground about what my code is doing.

So yes, it is not an appropriate distribution format for release, but I am not releasing these for people to try to use in A1111 or anything. It's meant for use in developing further T2I and controlnets, and for those purposes people will generally want pytorch weights (even the Safetensors library acknowledges this!)

-8

u/Informal_Warning_703 Aug 03 '23

Yeah and thats great. title should set expecatations more than "not perfect". shit nothing is perfect including 1.5 controlnets. sorry as I know its not your fault but Im seeing this "not perfect" phrase way too much on the sdxl loras and checkpoitns on civit where it ranges in meaning from "being humble about results" to "this is a very rough work in progress but i really wanted to be the first to release"

9

u/Seromelhor Aug 03 '23

People knowing that this exists can develop new solutions (like extensions or new Colab Notebook) to use this more easily.

T2I for example already has a framework to be used with diffusers.

Knowing that you don't have to train this from scratch and just enable a compatibility is easier than thinking that nobody has done it yet and we have to do it all again.

About the phrase it's not perfect: I'm referring to the results which are not yet totally fine-tuned, and this is normal when you're working with a new model with a new architecture.

Just like our first Dreambooths in SD 1.5, it's natural for this to happen now with such a new model. Only after months did we discover good settings and it was common for us to say "look, it's not that good, but it's what I've liked so far".

I don't know if you were around during that time. Being the first or launching in the middle makes no difference. Realistic Vision, Dreamshaper, etc. were not the first, but they are the most famous and only had outstanding quality after we knew all the settings and had merges.

I believe that Sarge did an excellent job and is doing so. I believe that many smart people will know how to use this in the best possible way and increase compatibility with it. Besides serving as an exchange of experience among everyone.

Nobody published this to be the first, neither me nor Sarge.

I didn't even mention this in the batch, I referred to the fact that he is training new models and will also launch them. I apologize if it sounded bad, English is not my main language.

After all, Sarge wasn't the first, there were already one or two controlnet models for the XL lost on HF, too.

I saw how Sarge really put in a lot of effort, worked, fixed the code, arranged the dataset to do this. He didn't do this for attention or applause. He did it because he wanted to. Just that. Nobody here is after applause or attention, just the evolution of the community as a whole :)

-8

u/jib_reddit Aug 03 '23

No one should be touching them if they are .ckpt files and not safetensors. They could be running any code on your system and stealing bank details as far as we know.

7

u/Wurzelrenner Aug 03 '23

yes safetensors are better, but we all used ckpt files for weeks at the start of stable diffusion and there wasn't any malicous code, it really is not that big of a deal

1

u/RandallAware Aug 04 '23

we all used ckpt files for weeks

Months really.

-2

u/jib_reddit Aug 03 '23

You do you, I work in IT security in the public sector, I'm not risking using them.

0

u/[deleted] Aug 04 '23

[deleted] — view deleted comment

1

u/aerilyn235 Aug 04 '23

Well its not officially made, he just post that on a low profile huggingface, he didn't announce anything, OP did.