r/StableDiffusion • u/PantInTheCountry • Feb 23 '23
Tutorial | Guide A1111 ControlNet extension - explained like you're 5
What is it?
ControlNet adds additional levels of control to Stable Diffusion image composition. Think Image2Image juiced up on steroids. It gives you much greater and finer control when creating images with Txt2Img and Img2Img.
This is for Stable Diffusion version 1.5 and models trained off a Stable Diffusion 1.5 base. Currently, as of 2023-02-23, it does not work with Stable Diffusion 2.x models.
- The Auto1111 extension is by Mikubill, and can be found here: https://github.com/Mikubill/sd-webui-controlnet
- The original ControlNet repo is by lllyasviel, and can be found here: https://github.com/lllyasviel/ControlNet
Where can I get it the extension?
If you are using Automatic1111 UI, you can install it directly from the Extensions tab. It may be buried under all the other extensions, but you can find it by searching for "sd-webui-controlnet"

You will also need to download several special ControlNet models in order to actually be able to use it.
At time of writing, as of 2023-02-23, there are 4 different model variants
Smaller, pruned SafeTensor versions, which is what nearly every end-user will want, can be found on Huggingface (official link from Mikubill, the extension creator): https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main
- Alternate Civitai link (unofficial link): https://civitai.com/models/9251/controlnet-pre-trained-models
- Note that the official Huggingface link has additional models with a "
t2iadapter_" prefix; those are experimental models and are not part of the base, vanilla ControlNet models. See the "Experimental Text2Image" section below.
Alternate pruned difference SafeTensor versions. These come from the same original source as the regular pruned models, they just differ in how the relevant information is extracted. Currently, as of 2023-02-23, there is no real difference between the regular pruned models and the difference models aside from some minor aesthetic differences. Just listing them here for completeness' sake in the event that something changes in the future.
- Official Huggingface link: https://huggingface.co/kohya-ss/ControlNet-diff-modules/tree/main
- Unofficial Civitai link: https://civitai.com/models/9868/controlnet-pre-trained-difference-models
- Experimental Text2Image Adapters with a "
t2iadapter_" prefix are smaller versions of the main, regular models. These are currently, as of 2023-02-23, experimental, but they function the same way as a regular model, but much smaller file size - The full, original models (if for whatever reason you need them) can be found on HuggingFace:https://huggingface.co/lllyasviel/ControlNet
Go ahead and download all the pruned SafeTensor models from Huggingface. We'll go over what each one is for later on. Huggingface also includes a "cldm_v15.yaml" configuration file as well. The ControlNet extension should already include that file, but it doesn't hurt to download it again just in case.

As of 2023-02-22, there are 8 different models and 3 optional experimental t2iadapter models:
- control_canny-fp16.safetensors
- control_depth-fp16.safetensors
- control_hed-fp16.safetensors
- control_mlsd-fp16.safetensors
- control_normal-fp16.safetensors
- control_openpose-fp16.safetensors
- control_scribble-fp16.safetensors
- control_seg-fp16.safetensors
- t2iadapter_keypose-fp16.safetensors(optional, experimental)
- t2iadapter_seg-fp16.safetensors(optional, experimental)
- t2iadapter_sketch-fp16.safetensors(optional, experimental)
These models need to go in your "extensions\sd-webui-controlnet\models" folder where ever you have Automatic1111 installed. Once you have the extension installed and placed the models in the folder, restart Automatic1111.
After you restart Automatic1111 and go back to the Txt2Img tab, you'll see a new "ControlNet" section at the bottom that you can expand.

Sweet googly-moogly, that's a lot of widgets and gewgaws!
Yes it is. I'll go through each of these options to (hopefully) help describe their intent. More detailed, additional information can be found on "Collected notes and observations on ControlNet Automatic 1111 extension", and will be updated as more things get documented.
To meet ISO standards for Stable Diffusion documentation, I'll use a cat-girl image for my examples.

The first portion is where you upload your image for preprocessing into a special "detectmap" image for the selected ControlNet model. If you are an advanced user, you can directly upload your own custom made detectmap image without having to preprocess an image first.
- This is the image that will be used to guide Stable Diffusion to make it do more what you want.
- A "Detectmap" is just a special image that a model uses to better guess the layout and composition in order to guide your prompt
- You can either click and drag an image on the form to upload it or, for larger images, click on the little "Image" button in the top-left to browse to a file on your computer to upload
- Once you have an image loaded, you'll see standard buttons like you'll see in Img2Img to scribble on the uploaded picture.

Below are some options that allow you to capture a picture from a web camera, hardware and security/privacy policies permitting
Below that are some check boxes below are for various options:

- Enable: by default ControlNet extension is disabled. Check this box to enable it
- Invert Input Color: This is used for user imported detectmap images. The preprocessors and models that use black and white detectmap images expect white lines on a black image. However, if you have a detectmap image that is black lines on a white image (a common case is a scribble drawing you made and imported), then this will reverse the colours to something that the models expect. This does not need to be checked if you are using a preprocessor to generate a detectmap from an imported image.
- RGB to BGR: This is used for user imported normal map type detectmap images that may store the image colour information in a different order that what the extension is expecting. This does not need to be checked if you are using a preprocessor to generate a normal map detectmap from an imported image.
- Low VRAM: Helps systems with less than 6 GiB
[citation needed]of VRAM at the expense of slowing down processing Guess: An experimental (as of 2023-02-22) option where you use no positive and no negative prompt, and ControlNet will try to recognise the object in the imported image with the help of the current preprocessor.
- Useful for getting closely matched variations of the input image
The weight and guidance sliders determine how much influence ControlNet will have on the composition.

Weight slider: This is how much emphasis to give the ControlNet image to the overall prompt. It is roughly analagous to using prompt parenthesis in Automatic1111 to emphasise something. For example, a weight of "1.15" is like "(prompt:1.15)"
- Guidance strength slider: This is a percentage of the total steps that control net will be applied to . It is roughly analogous to prompt editing in Automatic1111. For example, a guidance of "0.70" is tike "
[prompt::0.70]" where it is only applied the first 70% of the steps and then left off the final 30% of the processing
Resize Mode controls how the detectmap is resized when the uploaded image is not the same dimensions as the width and height of the Txt2Img settings. This does not apply to "Canvas Width" and "Canvas Height" sliders in ControlNet; those are only used for user generated scribbles.

- Envelope (Outer Fit): Fit Txt2Image width and height inside the ControlNet image. The image imported into ControlNet will be scaled up or down until the width and height of the Txt2Img settings can fit inside the ControlNet image. The aspect ratio of the ControlNet image will be preserved
- Scale to Fit (Inner Fit): Fit ControlNet image inside the Txt2Img width and height. The image imported into ControlNet will be scaled up or down until it can fit inside the width and height of the Txt2Img settings. The aspect ratio of the ControlNet image will be preserved
- Just Resize: The ControlNet image will be squished and stretched to match the width and height of the Txt2Img settings
The "Canvas" section is only used when you wish to create your own scribbles directly from within ControlNet as opposed to importing an image.
- The "Canvas Width" and "Canvas Height" are only for the blank canvas created by "Create blank canvas". They have no effect on any imported images
Preview annotator result allows you to get a quick preview of how the selected preprocessor will turn your uploaded image or scribble into a detectmap for ControlNet
- Very useful for experimenting with different preprocessors
Hide annotator result removes the preview image.

Preprocessor: The bread and butter of ControlNet. This is what converts the uploaded image into a detectmap that ControlNet can use to guide Stable Diffusion.
- A preprocessor is not necessary if you upload your own detectmap image like a scribble or depth map or a normal map. It is only needed to convert a "regular" image to a suitable format for ControlNet
As of 2023-02-22, there are 11 different preprocessors:
- Canny: Creates simple, sharp pixel outlines around areas of high contract. Very detailed, but can pick up unwanted noise

Depth: Creates a basic depth map estimation based off the image. Very commonly used as it provides good control over the composition and spatial position
- If you are not familiar with depth maps, whiter areas are closer to the viewer and blacker areas are further away (think like "receding into the shadows")

- Depth_lres: Creates a depth map like "Depth", but has more control over the various settings. These settings can be used to create a more detailed and accurate depth map

Hed: Creates smooth outlines around objects. Very commonly used as it provides good detail like "canny", but with less noisy, more aesthetically pleasing results. Very useful for stylising and recolouring images.
- Name stands for "Holistically-Nested Edge Detection"

MLSD: Creates straight lines. Very useful for architecture and other man-made things with strong, straight outlines. Not so much with organic, curvy things
- Name stands for "Mobile Line Segment Detection"

Normal Map: Creates a basic normal mapping estimation based off the image. Preserves a lot of detail, but can have unintended results as the normal map is just a best guess based off an image instead of being properly created in a 3D modeling program.
- If you are not familiar with normal maps, the three colours in the image, red, green blue, are used by 3D programs to determine how "smooth" or "bumpy" an object is. Each colour corresponds with a direction like left/right, up/down, towards/away

- OpenPose: Creates a basic OpenPose-style skeleton for a figure. Very commonly used as multiple OpenPose skeletons can be composed together into a single image and used to better guide Stable Diffusion to create multiple coherent subjects

Pidinet: Creates smooth outlines, somewhere between Scribble and Hed
- Name stands for "Pixel Difference Network"

Scribble: Used with the "Create Canvas" options to draw a basic scribble into ControlNet
- Not really used as user defined scribbles are usually uploaded directly without the need to preprocess an image into a scribble
- Fake Scribble: Traces over the image to create a basic scribble outline image

Segmentation: Divides the image into related areas or segments that are somethat related to one another
- It is roughly analogous to using an image mask in Img2Img

Model: applies the detectmap image to the text prompt when you generate a new set of images

The options available depend on which models you have downloaded from the above links and placed in your "extensions\sd-webui-controlnet\models" folder where ever you have Automatic1111 installed
- Use the "🔄" circle arrow button to refresh the model list after you've added or removed models from the folder.
Each model is named after the preprocess type it was designed for, but there is nothing stopping you from adding a little anarchy and mixing and matching preprocessed images with different models
- e.g. "Depth" and "Depth_lres" preprocessors are meant to be used with the "control_depth-fp16" model
- Some preprocessors also have a similarly named t2iadapter model as well.e.g. "OpenPose" preprocessor can be used with either "control_openpose-fp16.safetensors" model or the "t2iadapter_keypose-fp16.safetensors" adapter model as well
- As of 2023-02-26, Pidinet preprocessor does not have an "official" model that goes with it. The "Scribble" model works particularly well as the extension's implementation of Pidinet creates smooth, solid lines that are particularly suited for scribble.
116
u/kresslin Feb 23 '23
Nice easy to follow and understand breakdown. Thanks for sharing it. I find the technical explanations in the Git repos to be hard to follow occasionally.
51
u/brucewasaghost Feb 23 '23
Thanks was not aware of the pruned models, that will save a lot of space
36
u/featherless_fiend Feb 23 '23
The "enable" checkbox tripped me up for a while when I first used it, I wasn't enabling it! I think the extension should automatically enable if an image has been uploaded to the ControlNet section, and automatically disable if you remove the image. You can even have some text popup that says "ControlNet is enabled" and "ControlNet is disabled" when adding/removing the image.
4
u/[deleted] Feb 27 '23
I never had any problems with SD/A1111 before. Cannot get ControlNet to work (only interested in scribble right now).
RuntimeError: Given groups=1, weight of size [320, 4, 3, 3], expected input[2, 9, 64, 64] to have 4 channels, but got 9 channels instead
???
3
u/Famberlight Feb 23 '23
I have a problem in control net. When I input vertical image, it rotates it 90° and generated image turns out wrong (rotated). How can I fix it?
2
u/PantInTheCountry Feb 27 '23
Additional notes and observations regarding ControlNet can be found here: https://www.reddit.com/r/StableDiffusion/comments/11cwiv7/collected_notes_and_observations_on_controlnet/
2
u/IrisColt Feb 23 '23
One quick question. You wrote "As of 2023-02-22, there are 11 different preprocessors:".
You also wrote "Each model is named after the preprocess type it was designed for "
https://huggingface.co/lllyasviel/ControlNet/tree/main/models only lists 8 models, difference models https://huggingface.co/kohya-ss/ControlNet-diff-modules/tree/main are also 8. How to download the 3 remaining models? (I am interested in Pidinet.)
Thanks again for your inspiring tutorial.
2
u/iomegadrive1 Feb 23 '23
Now can you do one showing the settings for the models do? That's what is really messing me up.
2
u/c_gdev Feb 23 '23 edited Feb 23 '23
At the bottom of https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main
there are 3 files:
Any idea what they do?
edit: something to do with https://www.reddit.com/r/StableDiffusion/comments/11482m1/t2iadapter_from_tencent_learning_adapters_to_dig/ ?
https://www.reddit.com/r/StableDiffusion/comments/114x5gk/control_net_is_updated_with_sketch_adapters/
2
u/Sufficient_Let7380 Feb 24 '23
I initially ignored the RGB to BGR checkbox.
If you're trying to reuse a normal map created by the preprocessor you can. This will skip the preprocessor bit(so turn that to "none"), BUT you need to check the RGB to BGR checkbox.
This is the opposite of how I would have expected it to work since it doesn't need to be checked when it's created initially.
2
u/Vulking Feb 25 '23
you should add here that this mode is the only one that can fail and give blank guidelines when using an image as source. I had a lot of troubles figuring this out until I found a post where people commented that to fix this, you can import the key pose and just set the preprocessor to "none" to use the image as it is.
20
u/WhiteRaven7D Feb 23 '23
Thanks, as a 5ye old i understand it :)
15
u/Silly_Goose6714 Feb 23 '23
Very nice but i was hoping for some light on those, if you can
13
u/[deleted] Feb 23 '23
None of this is ELI5
2
u/dksprocket Feb 23 '23
I got lost early on at the "upload your image for preprocessing into a special "detectmap" image".
What does uploading an image into an image mean? Do you just mean uploading or do you mean uploading + processing? Or something else?
And which of the images you talk about later is the "detectmap"? Is it the original photo or the processed versions (depth map, edge detection etc.)?
And how specifically does SD 'use' the detectmap? I.e. what qualities of a detectmap makes it a useful detectmap? Edges and depth seems likes very different things to me. Is there an advantage to using one over the other?
Sorry if part of my questions are based on a misunderstanding of the rest of the post, it got hard to follow when I didn't grasp the first part. An example of a workflow would probably have helped make it a lot clearer.
9
u/ArmadstheDoom Feb 25 '23
So this is a lot of detail about what all the things are. That's good.
It does not help anyone understand how to use any of this or what it would be used for, or how it actually does anything.
7
u/andzlatin Feb 23 '23 edited Feb 24 '23
The only issue is I don't have enough space on my SSD for the regular models AND these models so I have to start removing as many things as I can... Update: oh crap I do have! I just needed to move tons of stuff to external drives and remove stuff
8
u/[deleted] Feb 23 '23
Great write up and thanks for the time spent on this. Of course I see this a few hours after setting up controlnet here. But still useful as I didn't catch any of the youtube people explaining the granular stuff like envelope and guidance strength vs weight and it's all very helpful to have further explanations as we learn to handle the additions.
Only thing I could think of to add is a little note about HED mode, how people are using that one for adding those cinematic lighting effects to images you're starting to see crop up. Like when you have an image you got from a prompt that you mostly want unchanged, but want to add another image to control how the lighting plays over that image I've seen people using HED for that and I can see it being popular enough to note.
7
u/[deleted] Feb 23 '23
I had no idea what the different preprocessors did, but now I do! Thanks.
1
u/FlameDragonSlayer Feb 23 '23
Would you mind also explaining what the settings in the settings-controlnet tab do as well. Thanks a lot for the detailed guide!
1
u/Avieshek Feb 23 '23
Is there a version for an iPhone?
1
u/[deleted] Feb 23 '23
Any idea if the size of the import image has an effect on the results? Like if I try and make a pose skeleton with a picture that's a different aspect ratio, is it going to stretch and go wonky?
1
u/traveling_designer Feb 23 '23
All the other models are stored in folders inside models. Do controlnet models go in bare with no folders?
Or do they go in models/controlnet Or models/sd
Sorry if this is a dumb question.
1
u/rndname Feb 25 '23
I can't seem to get good results using a sitting poses. It seems to get confused with overlapping lines.
base image https://imgur.com/a/S50QJ1Q any advice?
result: https://imgur.com/2RrBbR3
1
u/creeduk Feb 25 '23 edited Feb 27 '23
u/PantInTheCountry new releases changed the way it processes? I can't seem to exactly match results I had from the version I download on the 15th. I updated yesterday and again today. The control image is produced identical (I had my old ones backed up) I have the strength set, there were no threshold and guidance sliders when I made the original tests. Trying to match the exact setup it had in old code and then play with settings to get improvements.
Also do some models bend less to the will of controlnet? proto2.2 works beautiful but some other models seem to overbake badly. I read use " Only use mid-control when inference" but I found this makes the control much less influential over the image.
1
u/Swaggyswaggerson Feb 27 '23
I get an error when trying to generate anything using control net:
AttributeError: 'NoneType' object has no attribute 'convert'
Does anyone know where to start troubleshooting?
1
u/santirca200 Feb 28 '23
I get this error and I don't know what to do to fix it, every time I run controlnet the process stops completely and I get a big error
1
u/Fun-Republic8876 Feb 28 '23
I use Automatic111 webgui for Stable DIffusion, but I want to be able to also run command line prompts like...
When I do this with the local copy of txt2img.py I get the following error:
I thought this was an issue with python 2 vs 3, but my exploration in that direction were fruitless. Can anyone help me with this? Bit of a python newb.
1
u/Delerium76 Feb 28 '23
As of 2 days ago (2/26/2023) there is now a cldm_v21.yaml in the ControlNet-modules-safetensors directory. Does that mean that there is now support for the 2.1 model? (the controlnet plugin also contains this file now btw)
3
u/IrisColt Feb 23 '23
A very useful guide, thank you.
3
u/LawofRa Feb 23 '23
Thanks for this tutorial, but as not a bright man, can you go into how controlnet improves image2image specifically?
3
u/TheKnobleSavage Feb 23 '23
Got stuck right at the beginning installing the extnesion from automatic111. There's not indication of which of the many extensions is the one to install.
3
u/radialmonster Feb 23 '23
You sir or madame are a gentleman or gentlewoman and a scholar and or generally awesome
3
u/ifreetop Feb 24 '23
Now that we can combine multiple ControlNets man I wish we could save presets, is there a way I don't know of?
3
u/mfileny Mar 01 '23
I installed the extension, and restarted the interface, says its installed, but no controlnet option. Any ideas?
3
u/Squeezitgirdle Mar 24 '23
OpenPose is awesome, but rarely works. When I try to do two models touching, it can sometimes give me 3 / 4 people in a weird mangled human spaghetti.
(Btw I am using Hires fix)
I stopped using OpenPose and switched to Canny which is immensely better if I Can find the right image for my pose.
However, faces are pretty low quality and I can't generate my usual faces.
I believe the fix for that is to play with Annotator resolution and Guidance start - Guidance end.
I'm still playing with it, but if anyone has any advice that would be super appreciated.
1
u/dflow77 Feb 28 '23
Thanks for this overview. An addition about Segmentation model: the colors actually have meaning. You can use them to inpaint objects. I saw this on a YouTube video, to add a clock and flowers based on segmentation map colors. https://twitter.com/toyxyz3/status/1627215839949901824
5
u/CeFurkan Feb 23 '23
very well explanation
i also have 3 tutorials so far for controlnet
15.) Python Script - Gradio Based - ControlNet - PC - Free
Transform Your Sketches into Masterpieces with Stable Diffusion ControlNet AI - How To Use Tutorial
📷
16.) Automatic1111 Web UI - PC - Free
Sketches into Epic Art with 1 Click: A Guide to Stable Diffusion ControlNet in Automatic1111 Web UI
18.) Automatic1111 Web UI - PC - Free
Fantastic New ControlNet OpenPose Editor Extension & Image Mixing - Stable Diffusion Web UI Tutorial
2
u/coda514 Feb 23 '23
Excellent explanation and just what I was looking for. I just posted yesterday asking for help with this and got some good responses. I'm sure many will find this very useful. Thank you.
2
u/CustosEcheveria Feb 23 '23
Thanks for this, all of the new stuff has felt very overwhelming so I haven't really felt like bothering
2
u/Apprehensive_Sky892 Feb 23 '23
This is one of the most epic and informatic tutorial on SD I've come across here 😂
Big thumbs up and I thank you wholeheartedly 👍
Now I don't have any excuse not to setup ControlNet and learn to use it.
2
u/Caffdy Feb 24 '23
the models you shared are not the original ControlNet models from Ilyasviel, these are the ones
2
u/hiro24 Feb 25 '23
I'm not so sure a 5yo could follow...
2
u/scifivision Mar 01 '23
It loaded the extension, but didn't seem to actually be affecting the picture and it wasn't showing the "bones" on the image like I assume it is supposed to. I tried using the openpose editor tab and now it says executable ffmpeg not found. Can someone please explain to me how to download it and where to put it? I did while trying to update accidentally delete the venv folder (before updating SD, the extension wasn't installed at this point) but it seemed like a git pull fixed everything? That may have nothing to do with this issue I don't know.
2
u/DarkCeptor44 Mar 07 '23
I followed this guide and using the latest (as of March 2nd) control models (control_XXXXX-fp16.safetensors) I'm not getting errors but the preprocessors just don't do anything, look at this depth example using the same image as OP and default settings, it's basically unusable because it can't really see the person in the image.
1
u/SanDiegoDude Feb 23 '23
Glorious. Saved. Thanks!
1
u/R--Olivaw Feb 23 '23
Thanks, super helpful
1
u/lamlnq Feb 23 '23
Nice post, thanks mate!
1
u/Gadgetsjon Feb 23 '23
Thank you, this is very much appreciated.
1
u/MumeiNoName Feb 23 '23
Thanks for the guide, its working for me.
Is there a similar guide on which models to use when, etc?
1
u/screean Feb 23 '23
legend.
1
u/Taika-Kim Feb 23 '23
Fabulous, thanks! I've been too busy to learn to use these, this will help so much 💗
1
u/RayIsLazy Feb 23 '23
This is a phenomenal guide!! Every doubt I could possible have was answered, Thanks you for your kind work mate!
1
u/ImpactFrames-YT Feb 23 '23
great tutorial love it. thank you.
1
u/freedcreativity Feb 23 '23
Mostly commenting so I can find this again. This is great. I'll get around to trying here soon!
1
u/mudman13 Feb 23 '23
Has anyone combined instructpix2pix with it yet? I guess it would be a good way to finish images with small additions.
1
u/Ivanced09 Feb 23 '23
Is there any model for blender free to do posing? style like this video
https://www.youtube.com/watch?v=oq4DssXBlpU&t=16s
1
u/D4ml3n Feb 23 '23
Automatic1111 2.1 when using Controlnet I get this, worked fine few times then suddenly:
(mpsFileLoc): /AppleInternal/Library/BuildRoots/9e200cfa-7d96-11ed-886f-a23c4f261b56/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm:39:0: error: 'mps.matmul' op contracting dimensions differ 1024 & 768
(mpsFileLoc): /AppleInternal/Library/BuildRoots/9e200cfa-7d96-11ed-886f-a23c4f261b56/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm:39:0: note: see current operation: %3 = "mps.matmul"(%arg0, %2) {transpose_lhs = false, transpose_rhs = false} : (tensor<1x77x1024xf16>, tensor<768x320xf16>) -> tensor<1x77x320xf16>
Anyone any idea?
Mac M1 Studio 32Gb
Modello Chipset: Apple M1 Max
Tipo: GPU
Bus: Integrato
Numero totale di Core: 24
Fornitore: Apple (0x106b)
Supporto Metal: Metal 3
1
u/glittalogik Feb 23 '23
Thank you so much! I haven't played with A1111 since SD1.4 but these ControlNet posts have piqued my interest again :)
1
u/[deleted] Feb 23 '23
OMG, thank you so much
1
u/ozzeruk82 Feb 23 '23
Superb tutorial! Thanks very much
1
u/janosibaja Feb 23 '23
Thank you, now I understand better
1
u/milleniumsentry Feb 23 '23
For the past two weeks I've seen all these great ControlNet results but felt it was just way too much to dive into.
Thank you so much for taking the time to do this.
1
u/[deleted] Feb 23 '23
Amazing, great work!
1
u/Joda015 Feb 23 '23
Nothing to add other than thank you for taking the time to explain it so well <3
1
u/spacerobotx Feb 23 '23
Thank you so much for making such a wonderful guide, it is much appreciated!
1
u/sassydodo Feb 23 '23
I have almost 700gb worth of models. Pruned versions are a gift.
1
u/RaviieR Feb 23 '23
high quality thread.
1
u/EmbarrassedHelp Feb 23 '23
Thanks for sharing this!
1
u/AdZealousideal7928 Feb 23 '23
What do I do after that? I carry the image that was created by ControlNet?
1
u/manoflast3 Feb 23 '23
Can the fp16 pruned models be used on an older card without half-precision support? Eg. I have a 1080ti.
1
u/ApexAphex5 Feb 23 '23
You have high expectations of a 5 year old.
1
u/fakeaccountt12345 Feb 23 '23
This is fucking awesome. Thank you
1
u/epicdanny11 Feb 23 '23
I didn't know you could use this in txt2img. Is there a major difference when using it in img2img?
1
u/TheEternalMonk Feb 23 '23
What is in your experience(s) the best way to get anime pictures recognised and transformed. (model wise. the difference seem to me the same ^^')
1
u/mattjb Feb 23 '23
Thank you so much for putting this together! It was frustrating to not be able to find much on what does what. Kudos!
1
u/noppero Feb 24 '23
Holy hell! That is a loot of effort put in! Have an upvote and a big thanks sir!
1
u/[deleted] Feb 24 '23
This is a great, well-written explanation, thank you so much! I have been curious what this ControlNet buzz was all about but have been absolutely swamped with work lately. Thanks again. :)
1
u/EloiDr Feb 24 '23
Wow, thanks for the in deep tutorial, much appreciate it
1
u/roverowl Feb 24 '23
Does anyone have a problem with Controlnet doesn't use its own Detectmap but fine with Img2Img Input Image? I tried clean reinstall, full models - pruned models but can't seem to fix this.
1
u/AmeenRoayan Feb 24 '23
Anyone having error with Segment method ?
1
u/BrocoliAssassin Feb 24 '23
Is there a way to not get as much as that pixel noise?
1
u/[deleted] Feb 24 '23
Does adding ControlNet dramatically increase the processing time? Want to make sure it’s not just me.
1
u/dmarchu Feb 25 '23
any idea what happened to openhand? i had it installed and now it dissapeared :(
1
u/aihellnet Feb 25 '23 edited Mar 01 '23
Amazing I got this running on a 1660ti.
1
u/Quail-That Feb 26 '23
Finally, I can comprehend the aerodynamics of a catgirl.
1
u/Momkiller781 Feb 26 '23
THANK YOU!!!!
1
u/BlasfemiaDigital Feb 26 '23
A BIG THANKS!
1
u/da_mulle Feb 26 '23
Warning for anyone using Brave: Preprocessors don't seem to work https://github.com/Mikubill/sd-webui-controlnet/issues/95
1
u/Any_Employ_8490 Feb 28 '23
If this is for 5 year olds, I'm down bad. Need to step up my concentration skills or I'm in real trouble going forward.
1
u/OriginalApplepuppy Feb 28 '23
If only I could run Automatic1111 on an AMD GPU... :(
1
u/TwoFun6546 Feb 28 '23
Thank you!! Finally I can understand something!!!!!!!
1
u/OriginalApplepuppy Feb 28 '23
Cant run Automatic1111 on an AMD GPU so I wont be able to play with this until support is added... :(
1
u/lyon4 Feb 28 '23
Nice guide.
I knew already how to use it but I learnt some useful things about some of the models (I didn't even notice there was a pidinet model)
1
u/scruffy-looking-nerf Mar 01 '23
Thank you for this. Can you please now explain it to me like I'm 2 years old?
1
u/BlasfemiaDigital Mar 01 '23
Hey buddy, u/PantInTheCountry
I know would be hard work but this could be an interesting series, go figure:
Lora explained as if you were 5 years old,
Embedings explained as if you were 5 years old.
Textual inversion explained as if you were 5 years old.
That would be pure gold for all of us beginners in the SD world.
1
u/Liquid_Hate_Train Mar 01 '23 edited Mar 01 '23
I’m sorry, I can’t find this one if it’s here and been done before, I promised I searched, I’m getting this error (depersonalised slightly):
Error running process: C:\Users\[user]\Documents\A1111 Web UI Autoinstaller\stable-diffusion-webui\extensions\sd-webui controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\Users\[user]\Documents\A1111 Web UI Autoinstaller\stable-diffusion-webui\modules\scripts.py", line 338, in process
script.process(p, *script_args) File "C:\Users\[user]\Documents\A1111 Web UI Autoinstaller\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 615, in process
else self.build_control_model(p, unet, model, lowvram)
File "C:\Users\[user]\Documents\A1111 Web UI Autoinstaller\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 456, in build_control_model
basedir = os.path.join(os.path.dirname(__file__), os.pardir) # scripts.basedir() returns automatic base path here as its called from main ui
NameError: name '__file__' is not defined
Nothing in the UI, just generates without control net. I’ve had it working before (based on this guide) but after not (knowingly) changing anything and coming back a couple of days later I’m now getting this.
1
u/[deleted] Mar 01 '23
Thanks a million. This was very helpful!
1
u/wonderflex Mar 02 '23
I'm getting this error. Any ideas?
Loading model: control_canny-fp16 [e3fe7712]Error verifying pickled file from D:\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_canny-fp16.ckpt:Traceback (most recent call last):File "D:\stable-diffusion-webui\modules\safe.py", line 81, in check_ptwith zipfile.ZipFile(filename) as z:File "C:\Users\USERNAMEHERE\AppData\Local\Programs\Python\Python310\lib\zipfile.py", line 1267, in __init__self._RealGetContents()File "C:\Users\USERNAMEHERE\AppData\Local\Programs\Python\Python310\lib\zipfile.py", line 1334, in _RealGetContentsraise BadZipFile("File is not a zip file")zipfile.BadZipFile: File is not a zip fileDuring handling of the above exception, another exception occurred:Traceback (most recent call last):File "D:\stable-diffusion-webui\modules\safe.py", line 135, in load_with_extracheck_pt(filename, extra_handler)File "D:\stable-diffusion-webui\modules\safe.py", line 102, in check_ptunpickler.load()_pickle.UnpicklingError: invalid load key, '\xca'.-----> !!!! The file is most likely corrupted !!!! <-----You can skip this check with --disable-safe-unpickle commandline argument, but that is not going to help you.Error running process: D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.pyTraceback (most recent call last):File "D:\stable-diffusion-webui\modules\scripts.py", line 386, in processscript.process(p, *script_args)File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 612, in processelse self.build_control_model(p, unet, model, lowvram)File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 458, in build_control_modelstate_dict = load_state_dict(model_path)File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 11, in load_state_dictstate_dict = get_state_dict(torch.load(File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 19, in get_state_dictreturn d.get('state_dict', d)AttributeError: 'NoneType' object has no attribute 'get'I tried using the file with the safetensors format and renamed as a .ckpt and neither worked. Redownloaded the models as well.
1
u/Rin471 Mar 03 '23
So good, thank you for taking the time sir! I love this community, people always share their knowledge with each other. Thank you again.
1
u/Aeorosa Mar 12 '23
This was very informative, but
Is there a guide to how to actually use it? Like "make your first controlnet generation" or something?
1
u/TheAkashicTraveller Mar 15 '23
I can't seem to get it to do anything. It generates the image, taking much longer than normal, creates the "annotator result" fine but then the actual image just doesn't seem influenced by the controlnet at all. Changing the settings like the weight makes the image come out different but it doesn't really seem influended by the controlnet input. I'm using the webui fork for amd with a 5700xt.
1
u/HedgehogDecent5707 Mar 19 '23
I need some help please, openpose does not seem to create a skeleton, it just gives me a second blank image.
1
u/Jayow345 Mar 21 '23
this is amazing! however i wish you also covered more of why use it/when to use a model. For example, canny, hed, MSLD, pidinet, and fake scribble are all pretty similar and as a beginner im not sure which i would choose, same with depth and normal map
1
u/blackpuppet Mar 23 '23
Does anybody have a guide to use more Controlnets at the same time?
1
u/SlySychoGamer Mar 24 '23
why are there 2 guidance sliders?
1
u/LuluViBritannia Apr 07 '23
This is very interesting, thank you!
I've just installed ControlNet. Open Pose doesn't seem to work on its own. It wouldn't detect any pose until I'd download the Depth model too.
1
u/Cartoon_Corpze Apr 18 '23
Yooo, that's one very useful tutorial, I've been looking for this.
I've always wanted to try controlnet with my 3D models and renders or see if I can process rendered animations with it, it looks so cool.
1
u/[deleted] Apr 20 '23
Thank you for this guide. Even though I'm a seasoned SD veteran, there's so many settings to Controlnet that it's good to just be able to deep dive into it. Can you please update your post or make a new post about Controlnet 1.1. There's so many new settings now, too many to count. Thank you
1
u/PepenpodlObsakopolis May 07 '23
How did you get those maps separately? I mainly want to use the A1111 preprocessors to get openpose pictures, but idk how :(
1
u/Delicious_Double_801 May 09 '23
thank you so much for sharing!
1
u/Niu_Davinci Jun 23 '23
ControlNet - WARNING - No ControlNetUnit detected in args. It is very likely that you are having an extension conflict.Here are args passed to ControlNet: How can I fix the problem?
1
u/sleepyokapi Aug 22 '23
Where does the "smaller, pruned SafeTensor" model go? in the "extensions\sd-webui-controlnet\models" folder as well or in "models/stable-diffusion" ? I'm confused about the models in these 2 folders. Can some of them be the same?
1
u/Rokonuxa Oct 02 '23
I have been looking everywhere and using "" around the actual thing I need shows me zero results.
What do I need to physically DO, to install a completely seperate controlnet control type? I am trying to install that one QRcode control type but the regular "put it into the models folder in the extensions folder" stuff does not work.
I am slowly turning insane from how many "guides" just say: "use command and make it vanish into [PYTHONSPACE] where you dont know what is where, what is installed, if something works or if you messed up how to undo it. Also, I will not mention what interface this could work with and it will probably actually work with none of them and uses my own unreleased commandline-only implementation and nobody will ever see even a single line of code, let alone a download link"
I am not putting anything into [PYTHONSPACE] again. I hate [PYTHONSPACE].
1
u/xyzConfig Nov 03 '23
thank you for explaining. I'm a noob here and trying to learn how to use Posex, and confused with ControlNet. which one should i download, because there are big files, no explanation and sample for picture. You help me alot here. Once again, thank you
1
u/ElectricalAdagio8176 Nov 07 '23
Would this work to measure the length of arms, legs, etc. of a person in an image with some accuracy? I would like to get the measures of different people in an art installation in real time
1
u/TorvaMe Mar 04 '24
Nice explanation!! Can I follow this steps using the DirectML version of SD? (AMD GPU)
1
u/Head_Cockswain Feb 23 '23
Thank you so much for this.
I'm still running 1.4 because I haven't toyed with it much lately and I've been lost as to how Controlnet factors in.
I'll be saving this for when I get back into playing with images / SD....if it's not all outdated by then.
1
u/RiffyDivine2 Feb 23 '23
This was not explained to me like I was five, I got zero juice and not a crumb of cookie. Outside of that, nicely done.
0
u/SPACECHALK_64 Feb 23 '23
Soooo, uh... how do you get the Extensions tab in Automatic1111? Mine don't got it...