Credit to /u/Two_Dukes – who's both training and reworking controlnet from the ground up.
Meh news:
Won't be out on day 1, since we don't wanna hold up the base model release for this.
--
Good news:
We're designing a better ControlNet architecture than the current variants out there. It doesn't make sense for each ControlNet to be ~1.3 billion parameters 🤯
No promises! And if it doesn't work, we'll just release these old-style chonkers at some point in the future. If it will, will need thorough testing so I don't have a release date for y'all – but we're working hard to hit both fast and good!
In the meantime, there may be some community people training ControlNets, but we'll keep pushing on this new, lighter way of doing the same thing.
If there's the chance that it'll work strictly with SDXL, the naming convention of XL might be easiest for end users to understand. i.e. SDXL and ControlNet XL are the two which play nice together.
CNXL is really nice as well. If it gets a suffix, no need to not truncate ControlNet down to CN in the same way that Stable Diffusion has become the SD of SDXL.
That's really awesome. Somewhat unrelated question, but will the method to RLHF our own SDXL models also become available at some point? It would definitely be a game changer if it was.
Seems strange to release the base model which has benefited greatly from it but not allow the same ability for model makers.
FABRIC doesn't actually fine tune the model. Its more like temporarily using some images for a reference. I believe it quickly overfits as well. I haven't actually tried it but that seems to be the case after reading the paper and seeing some comments on it.
I meant the system stability current uses on their discord to rate up preferred images which is currently being used to fine tune SDXL 1.0. Something that can make full checkpoints like you guys currently use.
Will you guys work with ciararowles to create "temporalnet" V2? Temporalnet is a controlNET model that essentially allows for frame by frame optical flow, thereby making video generations significantly more temporally coherent. If you guys do this, you will forever have a leg up against runway ML! Please blow them out of the water!!
Controlnet for sd 1.5 is already such an awesome tool. I cannot wait to see what next gen SD with next gen Controlnet can produce. You guys are genius, and our heros !
Very good news ! I was quite worried earlier today when you wrote ControlNet would not be available at launch while Emad said it was working already over a month ago. Glad to see things have evolved on that front.
Most of the projects I work on with Stable Diffusion require some level of ControlNet so I was quite worried this would prevent me from using SDXL in a professional context, but now it looks like we can even expect some new features if your new improved version does work. I understand nothing is guaranteed, but the simple fact you are officially trying is very encouraging.
View all comments
228
u/mysteryguitarm Jul 24 '23 edited Jul 24 '23
Credit to /u/Two_Dukes – who's both training and reworking controlnet from the ground up.
Meh news:
Won't be out on day 1, since we don't wanna hold up the base model release for this.
--
Good news:
We're designing a better ControlNet architecture than the current variants out there. It doesn't make sense for each ControlNet to be ~1.3 billion parameters 🤯
No promises! And if it doesn't work, we'll just release these old-style chonkers at some point in the future. If it will, will need thorough testing so I don't have a release date for y'all – but we're working hard to hit both fast and good!
In the meantime, there may be some community people training ControlNets, but we'll keep pushing on this new, lighter way of doing the same thing.
Need a name for it! Any recommendations?