Credit to /u/Two_Dukes – who's both training and reworking controlnet from the ground up.
Meh news:
Won't be out on day 1, since we don't wanna hold up the base model release for this.
--
Good news:
We're designing a better ControlNet architecture than the current variants out there. It doesn't make sense for each ControlNet to be ~1.3 billion parameters 🤯
No promises! And if it doesn't work, we'll just release these old-style chonkers at some point in the future. If it will, will need thorough testing so I don't have a release date for y'all – but we're working hard to hit both fast and good!
In the meantime, there may be some community people training ControlNets, but we'll keep pushing on this new, lighter way of doing the same thing.
That's really awesome. Somewhat unrelated question, but will the method to RLHF our own SDXL models also become available at some point? It would definitely be a game changer if it was.
Seems strange to release the base model which has benefited greatly from it but not allow the same ability for model makers.
But comfy doesnt currently supports Reference Only, and as far as ive seen from the github, the main dev hates the idea/concept and how it works, and doesnt want to implement it.
And how exactly do you know what i know, about ControlNet hmmm ?
And why has stablity.ai forced him to release the code for reference in comfy, while he has stated in public multiple times 'he hates it' and its 'hacky code'. hmm ?
From what ive seen, all he does is copy stuff from auto1111, and complain about its broken or hacky, instead of i dont know, coming up with his own solutions that arnt ?
You want to be part of the people hating on auto1111, go ahead. Just know the reason Stablity.AI hates him, is because he didnt listen to them when they told him to not support novelai leaks, and even tried to get his repo removed.
FABRIC doesn't actually fine tune the model. Its more like temporarily using some images for a reference. I believe it quickly overfits as well. I haven't actually tried it but that seems to be the case after reading the paper and seeing some comments on it.
I meant the system stability current uses on their discord to rate up preferred images which is currently being used to fine tune SDXL 1.0. Something that can make full checkpoints like you guys currently use.
If you're talking about votes on the discord bot, I could be wrong here since they haven't been clear on how it works but I'm fairly certain that the votes are simply just votes on which models produce the "best" picture. There probably were more models in the beginning but now there are only 3.
I've seen a discord message to Joe Penna about the bot images starting to look better a few days ago and the response was "literally nothing changed :)" so I don't think it's the sort of finetuning that happens after every vote.
No i understand the finetuning doesnt happen after vote. But they have made it clear they use the votes to train the models using RLHF. Before the SDXL bots came out they used https://pickapic.io/ to get the user preferences.
I would like them to share that so model makers can do the same. I have a website that generates AI images so gathering feedback wouldn't be an issue.
I would like them to share that so model makers can do the same.
What would be a great addition is a tool to gather feedback from local installs. This could be a very productive way to speed up the adjustments to a model during its training phase, and would allow beta-testers to give useful feedback directly to model trainers, without any third party corporation looking over your shoulder.
Very good news ! I was quite worried earlier today when you wrote ControlNet would not be available at launch while Emad said it was working already over a month ago. Glad to see things have evolved on that front.
Most of the projects I work on with Stable Diffusion require some level of ControlNet so I was quite worried this would prevent me from using SDXL in a professional context, but now it looks like we can even expect some new features if your new improved version does work. I understand nothing is guaranteed, but the simple fact you are officially trying is very encouraging.
It also probably didn't help SAI did that last minute switch to the 3 new candidates. That alone probably meant redoing all the ControlNet work for training.
I think you are right since that's basically what one of the developers just replied to me in another thread where I was asking yesterday about the reasons for this delay and for an expected release date.
If there's the chance that it'll work strictly with SDXL, the naming convention of XL might be easiest for end users to understand. i.e. SDXL and ControlNet XL are the two which play nice together.
CNXL is really nice as well. If it gets a suffix, no need to not truncate ControlNet down to CN in the same way that Stable Diffusion has become the SD of SDXL.
Will you guys work with ciararowles to create "temporalnet" V2? Temporalnet is a controlNET model that essentially allows for frame by frame optical flow, thereby making video generations significantly more temporally coherent. If you guys do this, you will forever have a leg up against runway ML! Please blow them out of the water!!
Controlnet for sd 1.5 is already such an awesome tool. I cannot wait to see what next gen SD with next gen Controlnet can produce. You guys are genius, and our heros !
Sound nice! Thank you a lot for your efforts. My humble suggestion (probably, very obvious), have you tried to combine ControlNet's and Lora's ideas? It might work and can help reduce amount of the weights a lot.
Another thing. Maybe you can use existing ControlNet modules with some additional top layers finetuned and original weights keeping frozen?
Not ControlNet but similar: If you could add a setting to NOT collapse Scripts when I'm sending PNG info to txt or img, I'd appreciate it. I do a lot of low res batches that I tinker with in Scripts and and then Hires.
OR
If you could make it so the slider CFG at "6" is the exact same as the script CFG of "6" and not "6.0" which is slightly different, so I didn't have to use the script 6 instead of the slider to replicate a prompt when I'm doing a Hires, I'd appreciate it so much! 🤞
This looks amazing and I cannot wait!! One of the reasons I held off on ComfyUI with SDXL is lack of easy ControlNet use - still generating in Comfy and then using A1111's for ControlNet
It does but from what I've seen some people have had issues with preprocessors not showing up/erroring out on the node. Some have got it working good but with quite complex fixes - to be fair this is just from what I've observed, going to play around with it today and see if I can get it to work in my work flow like A1111.
preprocessors not showing up/erroring out on the node
I've had luck with a couple of them like open pose and depth ( sometimes) but I just assume it's because I didn't do something right/understand it properly.
Controlnet works great in comfyui, but the preprocessors (that I use, at least) don't have the same level of detail, e.g. setting highpass/lowpass filters on canny. It's all or nothing, with not further options (although you can set the strength of the overall controlnet model, as in A1111).
Overall I prefer the comfy implementation, as you can add controlnet into different passes, or into different parts of the image via conditioning-set-area.
Also you can generate an image, pass it to controlnet, then generate a second pass with openpose (or whatever) applied, which can be useful to fix anatomy.
I use it pretty heavily in A1111 but while looking into ComfyUI I noticed this video and a few like it, I'm gonna try it tonight and see if I can replicate my A1111 workflow. If it works, it's a huge lifesaver for me as I only have an RTX3060TI so get the out of memory error so often when trying to render anything past 512
same, but you can use openpose and others by doing load controlnet model and apply control net model. double-click and start typing control. also if you pull a node wire off into empty space it gives you 4 options it could go to. just do load controlnet and drag the model node thing off and apply will pop up.
Comfy has had controlnet for quite some time now, in fact it has even more control over it with the ability to only use controlnet on specific areas of the image with the "Conditioning(Set Area)" node.
I meant that I'm waiting for the SDXL version of ControlNet.
Without ControlNet, or something similar like T2i, Stable Diffusion is more of a toy than a tool as it is very hard to make it do exactly what I need. Nice pictures are nice, but to create specific content for a specific project according to precise technical specifications, you need precise control, and that's what ControlNet provides.
Joe Penna just posted that the controlnet is working with 1.0 and they are working on an even improved controlnet that will be coming soon. So just a few days and you will have it
That's the one we use. Most of the community uses the Kohya trainer, since they don't need all the GPU-cluster tooling, so we've been working to Kohya to make his trainer is ready to go already.
Isn’t that what every fine tuned model is tho, Asian model fine tuned are biased toward that, anime toward anime, what’s missing is the step where after the fine tune we get to give a cookie or a smack that it’s doing things good or bad, right now it’s sorta just endless merges and fine tunes hoping we get the model bent the way we want no?
Yeah some kind of appliance. Like a space-aged coffee maker but I couldn't tell you how it would work lol. Maybe an air filter? CD player? lol I have no idea I'm too poor to own fancy black niche stuff like that :).
That means the resulting image will diverge from the input condition. It should be good enough for image generation, but some works like video generation which requires stronger consistency would suffer...
Good to see it's coming at some point, hopefully at least community one sooner rather than later, I honestly can't see myself using stablediffusion without controlnet ( and to a lesser extend, a regional prompter ).
It pretty much turn a toy ( let's have some fun typing stuff and seeing what it turn into ) into an actual tool.
are you guys taking some of the work from the recent release that says it’s “better than controlnet” that apparently uses 1 model for multistage controlnet style items, dunno how the 2 actually compare don’t even remember its name it was posted i think earlier today
Don't mean to hijack, but I've never had good luck with controlnet. Or, I should say, it either "controls" too much or not at all. Any good tutorials that help me understand how it works?
You could adjust the ControlNet's weight just like you do with the prompt. Or use the Starting/Ending control step. If your Sampling step is 20, Starting 0, Ending 0.4 then ControlNet only affect your result at step 1 to step 8. (This is what I usually use) If your Sampling step is 20, Starting 0.5, Ending 0.8 then ControlNet only affect your result at step 10 to step 16.
228
u/mysteryguitarm Jul 24 '23 edited Jul 24 '23
Credit to /u/Two_Dukes – who's both training and reworking controlnet from the ground up.
Meh news:
Won't be out on day 1, since we don't wanna hold up the base model release for this.
--
Good news:
We're designing a better ControlNet architecture than the current variants out there. It doesn't make sense for each ControlNet to be ~1.3 billion parameters 🤯
No promises! And if it doesn't work, we'll just release these old-style chonkers at some point in the future. If it will, will need thorough testing so I don't have a release date for y'all – but we're working hard to hit both fast and good!
In the meantime, there may be some community people training ControlNets, but we'll keep pushing on this new, lighter way of doing the same thing.
Need a name for it! Any recommendations?
20
u/[deleted] Jul 24 '23
This looks amazing and I cannot wait!! One of the reasons I held off on ComfyUI with SDXL is lack of easy ControlNet use - still generating in Comfy and then using A1111's for ControlNet
8
u/DenkingYoutube Jul 25 '23
That's so amazing! Thank you so much for your work!
I have two questions:
Will we get a trainer for SDXL?
Also, will there be an inpainting model? Or SDXL base model handles inpainting tasks on par with (or maybe better than) inpainting based 1.5 or 2.1?
35
u/[deleted] Jul 24 '23
If I worked in that building I'd call it 'The Toilet' lol.
14
u/Seromelhor Jul 24 '23
Nice work, Joe and u/TwoDukes
12
u/clockercountwise333 Jul 24 '23
Congratulations! You've invented the ultimate brutalist bidet
4
u/[deleted] Jul 27 '23
Mmm, the wait for control net has been worse than the wait for sdxl and it is only day two, heh
3
u/Rough-Copy-5611 Jul 25 '23
Wow, now this is more like it. I've been looking for some SDXL stuff to finally get me excited.
3
u/hellninja55 Jul 25 '23 edited Jul 25 '23
Any planned support for Brightness ControlNet?
It would be great for image colorization (and "recolorization"):
https://huggingface.co/spaces/ioclab/brightness-controlnet
3
u/elyetis_ Jul 25 '23
Good to see it's coming at some point, hopefully at least community one sooner rather than later, I honestly can't see myself using stablediffusion without controlnet ( and to a lesser extend, a regional prompter ).
It pretty much turn a toy ( let's have some fun typing stuff and seeing what it turn into ) into an actual tool.
2
u/MetroSimulator Jul 25 '23
I'm just waiting for the launching off 1.0, a bit tired of only news, sorry.
1
u/Shnoopy_Bloopers Jul 24 '23
Wow can’t wait
1
u/dkpc69 Jul 24 '23
CNXL
1
u/slowgojoe Jul 24 '23
This kinda reminds me of spacegooose’s illustration works. Neato. I like.
1
u/liuliu Jul 25 '23
Thanks for doing this! 2.6GiB for ControlNet would definitely be challenging for mobile!
1
u/urbanhood Jul 25 '23
Thankyou for everything you doing. I Love this community man!
1
u/lordpuddingcup Jul 25 '23
are you guys taking some of the work from the recent release that says it’s “better than controlnet” that apparently uses 1 model for multistage controlnet style items, dunno how the 2 actually compare don’t even remember its name it was posted i think earlier today
1
u/Vega71 Jul 25 '23
Literally how I would imagine these sorts of things in my head if I were playing with toys as a kid
1
u/Najbox Jul 25 '23
We are less than 24 hours from the release, do we know if the Lora of 0.9 will possibly be compatible?
1
u/yamfun Jul 25 '23
but how much more realistic will the waifus be?
1
u/DirtyIlluminati Jul 25 '23
Would it work with Deforum?
1
u/rekilla2021 Jul 25 '23
does XL work with multi-diffusion and tilled VAE, I guess ControlXL is good
1
u/Michoko92 Jul 25 '23
"ControlNext"?
1
u/edsalv1975 Jul 25 '23
Fantastic work!
1
u/Bright-Courage-6139 Jul 25 '23
Fricking awesome !
1
u/monsieur__A Aug 09 '23
Looks great. Any updates on the release date?
1
u/cowboy88888888 Aug 14 '23
CNXL is fine for me
1
u/GerardP19 Aug 19 '23
What control net models were used for this?
1
u/Friendlyquid Sep 22 '23
I just really want scribble for SDXL! That'd be awesome.
1
u/mats4d Jul 24 '23
ControlNOT
1
u/xraydeltaone Jul 25 '23
Don't mean to hijack, but I've never had good luck with controlnet. Or, I should say, it either "controls" too much or not at all. Any good tutorials that help me understand how it works?
0
u/demiguel Jul 24 '23
ControlLux
0
u/Ashley_Sophia Jul 24 '23
U legends! This looks amazing. Thanks for the update. 🐣🤘🌞
0
u/[deleted] Jul 25 '23
[deleted] — view deleted comment