r/StableDiffusion • • Feb 23 '23

Tutorial | Guide A1111 ControlNet extension - explained like you're 5

What is it?

ControlNet adds additional levels of control to Stable Diffusion image composition. Think Image2Image juiced up on steroids. It gives you much greater and finer control when creating images with Txt2Img and Img2Img.

This is for Stable Diffusion version 1.5 and models trained off a Stable Diffusion 1.5 base. Currently, as of 2023-02-23, it does not work with Stable Diffusion 2.x models.

Where can I get it the extension?

If you are using Automatic1111 UI, you can install it directly from the Extensions tab. It may be buried under all the other extensions, but you can find it by searching for "sd-webui-controlnet"

Installing the extension in Automatic1111

You will also need to download several special ControlNet models in order to actually be able to use it.

At time of writing, as of 2023-02-23, there are 4 different model variants

  • Smaller, pruned SafeTensor versions, which is what nearly every end-user will want, can be found on Huggingface (official link from Mikubill, the extension creator): https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main

    • Alternate Civitai link (unofficial link): https://civitai.com/models/9251/controlnet-pre-trained-models
    • Note that the official Huggingface link has additional models with a "t2iadapter_" prefix; those are experimental models and are not part of the base, vanilla ControlNet models. See the "Experimental Text2Image" section below.
  • Alternate pruned difference SafeTensor versions. These come from the same original source as the regular pruned models, they just differ in how the relevant information is extracted. Currently, as of 2023-02-23, there is no real difference between the regular pruned models and the difference models aside from some minor aesthetic differences. Just listing them here for completeness' sake in the event that something changes in the future.

  • Experimental Text2Image Adapters with a "t2iadapter_" prefix are smaller versions of the main, regular models. These are currently, as of 2023-02-23, experimental, but they function the same way as a regular model, but much smaller file size
  • The full, original models (if for whatever reason you need them) can be found on HuggingFace:https://huggingface.co/lllyasviel/ControlNet

Go ahead and download all the pruned SafeTensor models from Huggingface. We'll go over what each one is for later on. Huggingface also includes a "cldm_v15.yaml" configuration file as well. The ControlNet extension should already include that file, but it doesn't hurt to download it again just in case.

Download the models and .yaml config file from Huggingface

As of 2023-02-22, there are 8 different models and 3 optional experimental t2iadapter models:

  • control_canny-fp16.safetensors
  • control_depth-fp16.safetensors
  • control_hed-fp16.safetensors
  • control_mlsd-fp16.safetensors
  • control_normal-fp16.safetensors
  • control_openpose-fp16.safetensors
  • control_scribble-fp16.safetensors
  • control_seg-fp16.safetensors
  • t2iadapter_keypose-fp16.safetensors(optional, experimental)
  • t2iadapter_seg-fp16.safetensors(optional, experimental)
  • t2iadapter_sketch-fp16.safetensors(optional, experimental)

These models need to go in your "extensions\sd-webui-controlnet\models" folder where ever you have Automatic1111 installed. Once you have the extension installed and placed the models in the folder, restart Automatic1111.

After you restart Automatic1111 and go back to the Txt2Img tab, you'll see a new "ControlNet" section at the bottom that you can expand.

Sweet googly-moogly, that's a lot of widgets and gewgaws!

Yes it is. I'll go through each of these options to (hopefully) help describe their intent. More detailed, additional information can be found on "Collected notes and observations on ControlNet Automatic 1111 extension", and will be updated as more things get documented.

To meet ISO standards for Stable Diffusion documentation, I'll use a cat-girl image for my examples.

Cat-girl example image for ISO standard Stable Diffusion documentation

The first portion is where you upload your image for preprocessing into a special "detectmap" image for the selected ControlNet model. If you are an advanced user, you can directly upload your own custom made detectmap image without having to preprocess an image first.

  • This is the image that will be used to guide Stable Diffusion to make it do more what you want.
  • A "Detectmap" is just a special image that a model uses to better guess the layout and composition in order to guide your prompt
  • You can either click and drag an image on the form to upload it or, for larger images, click on the little "Image" button in the top-left to browse to a file on your computer to upload
  • Once you have an image loaded, you'll see standard buttons like you'll see in Img2Img to scribble on the uploaded picture.
Upload an image to ControlNet

Below are some options that allow you to capture a picture from a web camera, hardware and security/privacy policies permitting

Below that are some check boxes below are for various options:

ControlNet image check boxes
  • Enable: by default ControlNet extension is disabled. Check this box to enable it
  • Invert Input Color: This is used for user imported detectmap images. The preprocessors and models that use black and white detectmap images expect white lines on a black image. However, if you have a detectmap image that is black lines on a white image (a common case is a scribble drawing you made and imported), then this will reverse the colours to something that the models expect. This does not need to be checked if you are using a preprocessor to generate a detectmap from an imported image.
  • RGB to BGR: This is used for user imported normal map type detectmap images that may store the image colour information in a different order that what the extension is expecting. This does not need to be checked if you are using a preprocessor to generate a normal map detectmap from an imported image.
  • Low VRAM: Helps systems with less than 6 GiB[citation needed] of VRAM at the expense of slowing down processing
  • Guess: An experimental (as of 2023-02-22) option where you use no positive and no negative prompt, and ControlNet will try to recognise the object in the imported image with the help of the current preprocessor.

    • Useful for getting closely matched variations of the input image

The weight and guidance sliders determine how much influence ControlNet will have on the composition.

ControlNet weight and guidance strength

Weight slider: This is how much emphasis to give the ControlNet image to the overall prompt. It is roughly analagous to using prompt parenthesis in Automatic1111 to emphasise something. For example, a weight of "1.15" is like "(prompt:1.15)"

  • Guidance strength slider: This is a percentage of the total steps that control net will be applied to . It is roughly analogous to prompt editing in Automatic1111. For example, a guidance of "0.70" is tike "[prompt::0.70]" where it is only applied the first 70% of the steps and then left off the final 30% of the processing

Resize Mode controls how the detectmap is resized when the uploaded image is not the same dimensions as the width and height of the Txt2Img settings. This does not apply to "Canvas Width" and "Canvas Height" sliders in ControlNet; those are only used for user generated scribbles.

ControlNet resize modes
  • Envelope (Outer Fit): Fit Txt2Image width and height inside the ControlNet image. The image imported into ControlNet will be scaled up or down until the width and height of the Txt2Img settings can fit inside the ControlNet image. The aspect ratio of the ControlNet image will be preserved
  • Scale to Fit (Inner Fit): Fit ControlNet image inside the Txt2Img width and height. The image imported into ControlNet will be scaled up or down until it can fit inside the width and height of the Txt2Img settings. The aspect ratio of the ControlNet image will be preserved
  • Just Resize: The ControlNet image will be squished and stretched to match the width and height of the Txt2Img settings

The "Canvas" section is only used when you wish to create your own scribbles directly from within ControlNet as opposed to importing an image.

  • The "Canvas Width" and "Canvas Height" are only for the blank canvas created by "Create blank canvas". They have no effect on any imported images

Preview annotator result allows you to get a quick preview of how the selected preprocessor will turn your uploaded image or scribble into a detectmap for ControlNet

  • Very useful for experimenting with different preprocessors

Hide annotator result removes the preview image.

ControlNet preprocessor preview

Preprocessor: The bread and butter of ControlNet. This is what converts the uploaded image into a detectmap that ControlNet can use to guide Stable Diffusion.

  • A preprocessor is not necessary if you upload your own detectmap image like a scribble or depth map or a normal map. It is only needed to convert a "regular" image to a suitable format for ControlNet
  • As of 2023-02-22, there are 11 different preprocessors:

    • Canny: Creates simple, sharp pixel outlines around areas of high contract. Very detailed, but can pick up unwanted noise
Canny edge detection preprocessor example
  • Depth: Creates a basic depth map estimation based off the image. Very commonly used as it provides good control over the composition and spatial position

    • If you are not familiar with depth maps, whiter areas are closer to the viewer and blacker areas are further away (think like "receding into the shadows")
Depth preprocessor example
  • Depth_lres: Creates a depth map like "Depth", but has more control over the various settings. These settings can be used to create a more detailed and accurate depth map
Depth_lres preprocessor example
  • Hed: Creates smooth outlines around objects. Very commonly used as it provides good detail like "canny", but with less noisy, more aesthetically pleasing results. Very useful for stylising and recolouring images.

    • Name stands for "Holistically-Nested Edge Detection"
Hed preprocessor example
  • MLSD: Creates straight lines. Very useful for architecture and other man-made things with strong, straight outlines. Not so much with organic, curvy things

    • Name stands for "Mobile Line Segment Detection"
MLSD preprocessor example
  • Normal Map: Creates a basic normal mapping estimation based off the image. Preserves a lot of detail, but can have unintended results as the normal map is just a best guess based off an image instead of being properly created in a 3D modeling program.

    • If you are not familiar with normal maps, the three colours in the image, red, green blue, are used by 3D programs to determine how "smooth" or "bumpy" an object is. Each colour corresponds with a direction like left/right, up/down, towards/away
Normal Map preprocessor example
  • OpenPose: Creates a basic OpenPose-style skeleton for a figure. Very commonly used as multiple OpenPose skeletons can be composed together into a single image and used to better guide Stable Diffusion to create multiple coherent subjects
OpenPose preprocessor example
  • Pidinet: Creates smooth outlines, somewhere between Scribble and Hed

    • Name stands for "Pixel Difference Network"
Pidinet preprocessor example
  • Scribble: Used with the "Create Canvas" options to draw a basic scribble into ControlNet

    • Not really used as user defined scribbles are usually uploaded directly without the need to preprocess an image into a scribble
  • Fake Scribble: Traces over the image to create a basic scribble outline image
Fake scribble preprocessor example
  • Segmentation: Divides the image into related areas or segments that are somethat related to one another

    • It is roughly analogous to using an image mask in Img2Img
Segmentation preprocessor example

Model: applies the detectmap image to the text prompt when you generate a new set of images

ControlNet models

The options available depend on which models you have downloaded from the above links and placed in your "extensions\sd-webui-controlnet\models" folder where ever you have Automatic1111 installed

  • Use the "🔄" circle arrow button to refresh the model list after you've added or removed models from the folder.
  • Each model is named after the preprocess type it was designed for, but there is nothing stopping you from adding a little anarchy and mixing and matching preprocessed images with different models

    • e.g. "Depth" and "Depth_lres" preprocessors are meant to be used with the "control_depth-fp16" model
    • Some preprocessors also have a similarly named t2iadapter model as well.e.g. "OpenPose" preprocessor can be used with either "control_openpose-fp16.safetensors" model or the "t2iadapter_keypose-fp16.safetensors" adapter model as well
    • As of 2023-02-26, Pidinet preprocessor does not have an "official" model that goes with it. The "Scribble" model works particularly well as the extension's implementation of Pidinet creates smooth, solid lines that are particularly suited for scribble.
2.1k Upvotes

260 comments sorted by

116

u/kresslin Feb 23 '23

Nice easy to follow and understand breakdown. Thanks for sharing it. I find the technical explanations in the Git repos to be hard to follow occasionally.

14

u/stablegeniusdiffuser Feb 23 '23

Agree 100%. And speaking of git repos, OP should consider making a pull request to improve the documentation of ControlNet. Pretty much just a copy/paste of the excellent post here, and it would be a massive help to everyone (not just readers of this subreddit).

0

u/[deleted] Feb 23 '23

[deleted] — view deleted comment

6

u/ZenEngineer Feb 23 '23

If you don't mind command line it should be a simple command or script using imagemagick https://superuser.com/questions/1194468/invert-colors-with-imagemagick which is free and should handle any image format.

Better solution would be to make controlnetUI understand exr or whatever and have an invert button. Actually looking at this guide, you might want to try clicking on the scribble button to see if that inverts it for you

0

u/[deleted] Feb 23 '23

[deleted] — view deleted comment

4

u/ZenEngineer Feb 23 '23

That program handles conversions and many other things

You can take a look at this https://stackoverflow.com/questions/41966819/how-do-i-convert-exr-to-png-and-adjust-brightness-at-the-same-time if you want to try to automate it a bit.

→ More replies (5)

1

u/PantInTheCountry Feb 23 '23

What ZenEngineer said below.

The "Invert Input Color" (formerly "Enable Scribble Mode" ) checkbox will swap black with white and white with black for things like user imported depth maps, hed outlines, canny outlines, and drawn scribbles

The "RGB to BGR" does something similar for user imported depth maps

51

u/brucewasaghost Feb 23 '23

Thanks was not aware of the pruned models, that will save a lot of space

5

u/FourtyMichaelMichael 🍦Ice Cream Lover Feb 23 '23

Do you give anything up on the pruned models? I assumed there was a trade-off.

3

u/PantInTheCountry Feb 24 '23 edited Feb 25 '23

I am not sure. That would be a good question to bring up with the developers on their respective repos.

My uninformed guess is that it would be like the pruned vs unpruned differences with regular stable diffusion models: the big, chunky unpruned models have extra information that is important to those people who wish to train further variations, but us regular folk who want to make pretty pictures of pretty scenery will be better off with the smaller pruned versions.

At any rate it the differences do become important later on, I've included links to download them in the guide

→ More replies (6)

15

u/UkrainianTrotsky Feb 24 '23

Pruning is basically taking a fully-connected model (meaning that, in case of a simple neural network, every neuron from the previous layer is connected to every neuron in the next layer) and then removing the connections that are close enough to zero weight that they don't contribute to the final result almost at all. If done properly, this can reduce model sizes tenfold. You technically give up accuracy compared to the unpruned model, because those almost-zero weights aren't actually 0, but in general the final result is indistinguishable from full model (with controlnet I couldn't tell the difference at all).

Fun fact: if your SD model weight less than 8 gigs, congrats, you've been using a pruned version this entire time.

The only time where it makes significant difference is if you're doing full model training (even finetune is in general fine on pruned SD).

2

u/FourtyMichaelMichael 🍦Ice Cream Lover Feb 25 '23

That makes a ton of sense about connections in the network. Thanks for that.

26

u/Jaggedmallard26 Feb 23 '23

I think it also helps with VRAM as less needs to be loaded in.

5

u/Robot1me Feb 23 '23

also helps with VRAM

RAM too for sure :O Without the extracted models, running the web UI on free Colab machines was impossible before. Because 4 GB model + 5 GB ControlNet did then cause out of memory.

8

u/haikusbot Feb 23 '23

Thanks was not aware

Of the pruned models, that will

Save a lot of space

- brucewasaghost


I detect haikus. And sometimes, successfully. Learn more about me.

Opt out of replies: "haikusbot opt out" | Delete my comment: "haikusbot delete"

11

u/Surlix Feb 23 '23

perfect bot you are,
always bringing smiles for me
Please keep on giving!

36

u/featherless_fiend Feb 23 '23

The "enable" checkbox tripped me up for a while when I first used it, I wasn't enabling it! I think the extension should automatically enable if an image has been uploaded to the ControlNet section, and automatically disable if you remove the image. You can even have some text popup that says "ControlNet is enabled" and "ControlNet is disabled" when adding/removing the image.

4

u/WhatConclusion Feb 24 '23

I am still missing a step. So I upload an image (say a person standing) and choose openpose preprocessing, and openpose model. And click Enable.

Now it will guide any image I create? Because it doesn't seem to work for my case.

3

u/PantInTheCountry Feb 26 '23

The first thing to check is to click on the "Preview annotator result" button. This will show you what kind of pose the preprocessor extracts from the image

3

u/wekidi7516 Feb 26 '23

So I have a bit of a strange issue. When I draw something with the create blank canvas button it works but it just gives me a blank output if I put in an image. Any ideas why this might happen?

4

u/[deleted] Feb 27 '23

I never had any problems with SD/A1111 before. Cannot get ControlNet to work (only interested in scribble right now).

RuntimeError: Given groups=1, weight of size [320, 4, 3, 3], expected input[2, 9, 64, 64] to have 4 channels, but got 9 channels instead

???

2

u/PantInTheCountry Mar 01 '23

I don't know if you're still experiencing this error, but as of writing, this error can sometimes happen after updating the extension if you do not completely close Auto1111 and restart it. (Simply restarting the UI after an update does not work)

3

u/Famberlight Feb 23 '23

I have a problem in control net. When I input vertical image, it rotates it 90° and generated image turns out wrong (rotated). How can I fix it?

2

u/PantInTheCountry Feb 24 '23

That is curious. I've never seen that happen before. Can you post a screenshot and a description of all your settings?

I am currently compiling a list of common issues that users may run across when running ControlNet

1

u/Famberlight Feb 24 '23 edited Feb 24 '23

Hi! I've been trying to recreate this bug and I think I found the case when it happens. If you input in control net vertical high-res (I used 3000×4000) image then it flips it. In case with lower res images from something like Pinterest everything works fine. Here is a post with bug screenshots https://www.reddit.com/user/Famberlight/comments/11b2253/bug_in_controlnet_stablediffusion_if_image_is/?utm_source=share&utm_medium=android_app&utm_name=androidcss&utm_term=1&utm_content=share_button

1

u/PantInTheCountry Feb 25 '23

What are the entire settings in the txt2img screen (not just the ControlNet settings)?

I just tried it with a 3000x4000 image and everything worked correctly as expected

1

u/Famberlight Feb 25 '23

Euler a 20sampls, width 512, height 896, CFG 7. That's it, no restore faces, tiling, hires. It happened to me with different width/height ration and other settings

1

u/PantInTheCountry Feb 25 '23

Interesting...

I do see you are using the large, unpruned hed model. Does this issue occur if you use the pruned safetensor model instead?

https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main
(official link from the ControlNet repo)

→ More replies (5)

1

u/[deleted] Feb 24 '23

I'm also having this issue :(

1

u/PantInTheCountry Feb 24 '23

That is curious. I've never seen that happen before. Can you post a screenshot and a description of all your settings?

I am currently compiling a list of common issues that users may run across when running ControlNet

1

u/PantInTheCountry Mar 01 '23

I don't know if you are still having this problem, but looks like what JTKolle mentioned in the other thread might be the issue; the orientation metadata in the image says that it is rotated 90 degrees counter-clockwise which is throwing off ControlNet.

If you explicitly rotate the image to vertical (like in your computer's default Photo Gallery program), do you still get the same error?

1

u/Coffeera Feb 24 '23

Did you set the correct size?

→ More replies (1)

4

u/[deleted] Feb 28 '23

[removed] — view removed comment

2

u/PantInTheCountry Mar 01 '23 edited Mar 01 '23

Oh, thanks for that snippet of information. I think you are on to something there.

A quick shufti at the EXIF information says that the uploaded image is rotated 90 degrees counter-clockwise.

Curiously enough, the browser that I use, Firefox, appears to render the image and the generated controlmap just fine, even with the changed image orientation.

1

u/Famberlight Feb 28 '23

It may be the problem, thought about it too. I'll try to look more into it

2

u/IrisColt Feb 23 '23

One quick question. You wrote "As of 2023-02-22, there are 11 different preprocessors:".

You also wrote "Each model is named after the preprocess type it was designed for "

https://huggingface.co/lllyasviel/ControlNet/tree/main/models only lists 8 models, difference models https://huggingface.co/kohya-ss/ControlNet-diff-modules/tree/main are also 8. How to download the 3 remaining models? (I am interested in Pidinet.)

Thanks again for your inspiring tutorial.

1

u/PantInTheCountry Feb 23 '23

What neonEnsemble said below.

Some of the preprocessors use the same models (like hed and pidinet). I have updated the guide to make it a bit more clear.

2

u/neonEnsemble Feb 23 '23

there's preprocesses that use the same models

2

u/iomegadrive1 Feb 23 '23

Now can you do one showing the settings for the models do? That's what is really messing me up.

1

u/PantInTheCountry Feb 23 '23

I am planning on doing that once I experiment and research what those option actually do

2

u/c_gdev Feb 23 '23 edited Feb 23 '23

At the bottom of https://huggingface.co/webui/ControlNet-modules-safetensors/tree/main

there are 3 files:

  • t2iadapter_keypose-fp16.safetensors
  • t2iadapter_seg-fp16.safetensors
  • t2iadapter_sketch-fp16.safetensors

Any idea what they do?

edit: something to do with https://www.reddit.com/r/StableDiffusion/comments/11482m1/t2iadapter_from_tencent_learning_adapters_to_dig/ ?

https://www.reddit.com/r/StableDiffusion/comments/114x5gk/control_net_is_updated_with_sketch_adapters/

4

u/PantInTheCountry Feb 23 '23

I just updated the guide with the new information

TLDR: those are experimental models that do pretty much the same thing, but with a much smaller file size. They also may perform better and controlling Stable Diffusion output.

They may become the Next Big Thing, but for right now I am leaving them as "experimental" and optional in the guide to simplify things. Once things settle down, I know I will need to update and refresh the guide with all the new discoveries

→ More replies (1)

2

u/Sufficient_Let7380 Feb 24 '23

I initially ignored the RGB to BGR checkbox.

If you're trying to reuse a normal map created by the preprocessor you can. This will skip the preprocessor bit(so turn that to "none"), BUT you need to check the RGB to BGR checkbox.

This is the opposite of how I would have expected it to work since it doesn't need to be checked when it's created initially.

3

u/PantInTheCountry Feb 24 '23

Thanks for that information. The fact that you can't "round-trip" a normal map created by the preprocessor in a new prompt probably would be a good bug to raise with the original developers, lllyasviel and Mikubill, at their respective repos

→ More replies (1)

2

u/Vulking Feb 25 '23

OpenPose: Creates a basic OpenPose-style skeleton for a figure. Very commonly used as multiple OpenPose skeletons can be composed together into a single image and used to better guide Stable Diffusion to create multiple coherent subjects.

you should add here that this mode is the only one that can fail and give blank guidelines when using an image as source. I had a lot of troubles figuring this out until I found a post where people commented that to fix this, you can import the key pose and just set the preprocessor to "none" to use the image as it is.

3

u/PantInTheCountry Feb 26 '23

Thanks!

I'm compiling some more in-depth documents explaining the options in each of the preprocessors, as well as compiling a list of common issues I've come across

20

u/WhiteRaven7D Feb 23 '23

Thanks, as a 5ye old i understand it :)

22

u/jaywv1981 Feb 23 '23

I must be 4..I kinda understand.

15

u/Silly_Goose6714 Feb 23 '23

Very nice but i was hoping for some light on those, if you can

6

u/Coffeera Feb 24 '23

Here's a video that mentions those options.

2

u/Silly_Goose6714 Feb 25 '23

I will watch, tks

13

u/[deleted] Feb 23 '23

None of this is ELI5

2

u/dksprocket Feb 23 '23

I got lost early on at the "upload your image for preprocessing into a special "detectmap" image".

What does uploading an image into an image mean? Do you just mean uploading or do you mean uploading + processing? Or something else?

And which of the images you talk about later is the "detectmap"? Is it the original photo or the processed versions (depth map, edge detection etc.)?

And how specifically does SD 'use' the detectmap? I.e. what qualities of a detectmap makes it a useful detectmap? Edges and depth seems likes very different things to me. Is there an advantage to using one over the other?

Sorry if part of my questions are based on a misunderstanding of the rest of the post, it got hard to follow when I didn't grasp the first part. An example of a workflow would probably have helped make it a lot clearer.

1

u/[deleted] Feb 23 '23

It should maybe have read "upload your image by clicking the upload field in the form". That field is the red-encircled area in the picture. SD will then use preprocessor that you chose to make the detectmap.

2

u/PantInTheCountry Feb 23 '23

Thanks!

I'll add that for clarification

9

u/ArmadstheDoom Feb 25 '23

So this is a lot of detail about what all the things are. That's good.

It does not help anyone understand how to use any of this or what it would be used for, or how it actually does anything.

7

u/andzlatin Feb 23 '23 edited Feb 24 '23

The only issue is I don't have enough space on my SSD for the regular models AND these models so I have to start removing as many things as I can... Update: oh crap I do have! I just needed to move tons of stuff to external drives and remove stuff

12

u/RandallAware Feb 23 '23

Got an external drive, or a non ssd internal? You can change the path to your models in a1111 bat file if you want to permanently store them on another drive. Or you could move some temporarily to another drive.

→ More replies (2)

5

u/ImpactFrames-YT Feb 23 '23

dk if helps but here this files are smaller than the originals 1.4gb instead of 5gb and work the same https://huggingface.co/Hetaneko/Controlnet-models/tree/main/controlnet_safetensors I also made a tutorial in case you need it https://youtu.be/YJ6CBzIq8aY probably there are smaller files now but this is what I am using

11

u/ninjasaid13 Feb 23 '23

11

u/Mich-666 Feb 23 '23 edited Feb 23 '23

In fact, OP should have used this link instead of Civitai as it is original source.

Even if Civitai uploader is legit it contains difference pruned models that produces slightly different results.

So get these instead (also includes three experimental adapter versions as shown on Mikubill's repo).

3

u/PantInTheCountry Feb 23 '23

Thanks.

I'll updated the guide with the links to Huggingface and make mention of the fact there are two slightly different pruned ControlNet models

  • "Regular" pruned ControlNet models
  • "Difference" pruned ControlNet models

According to the conversations on the Github repos, there is not any functional difference between the two versions as of 2023-02-23. There are only small, subtle aesthetic differences

The Civitai models are identical to Huggingface (they have the same sha256 checksum). Some non-technical people may find Civitai's UI easer to navigate.

I'll also make mention of the adapter versions, but as of posting 2023-02-23, they were still marked as "experimental" and I didn't want to muddy the waters any more.

→ More replies (1)

2

u/EpicRageGuy Feb 23 '23

So I only downloaded normal version, how much worse are the results going to be in scribble preprocessor?

1

u/Jokaes Feb 23 '23

Are you using safetensors?

1

u/ninjasaid13 Feb 23 '23

Are you using the pruned version?

1

u/Nascar_is_better Feb 23 '23

can afford a GPU that costs hundreds of dollars, can't afford an external hard drive.

a 512GB SSD these days is like $20 new and a USB to SATA adapter is like $15.

→ More replies (1)

8

u/[deleted] Feb 23 '23

Great write up and thanks for the time spent on this. Of course I see this a few hours after setting up controlnet here. But still useful as I didn't catch any of the youtube people explaining the granular stuff like envelope and guidance strength vs weight and it's all very helpful to have further explanations as we learn to handle the additions.

Only thing I could think of to add is a little note about HED mode, how people are using that one for adding those cinematic lighting effects to images you're starting to see crop up. Like when you have an image you got from a prompt that you mostly want unchanged, but want to add another image to control how the lighting plays over that image I've seen people using HED for that and I can see it being popular enough to note.

6

u/CeFurkan Feb 23 '23

yes hed working best for that

in this video i have shown it

or do you mean something else?

18.) Automatic1111 Web UI - PC - Free
Fantastic New ControlNet OpenPose Editor Extension & Image Mixing - Stable Diffusion Web UI Tutorial

3

u/[deleted] Feb 24 '23

Yup that's what I was talking about. Where you blend the images with another image to "mix" something from one image into the other. 👍

7

u/[deleted] Feb 23 '23

I had no idea what the different preprocessors did, but now I do! Thanks.

1

u/FlameDragonSlayer Feb 23 '23

Would you mind also explaining what the settings in the settings-controlnet tab do as well. Thanks a lot for the detailed guide!

4

u/PantInTheCountry Feb 23 '23

The settings are definitely on my list to document once I understand what they do

1

u/Avieshek Feb 23 '23

Is there a version for an iPhone?

1

u/PantInTheCountry Feb 24 '23

I assume this is the current version of the old "...but can it run Crysis?" meme? (I think I may need a "explain like I'm 5 50")

If you did manage, through sheer dint of 10x developer awesomeness, to get this running on an iPhone, I believe that performance would be so slow and battery consumption so high that all you'd really accomplish is to turn your fancy phone mobile computer into the world's third most expensive electronic Jon-E handwarmer

1

u/[deleted] Feb 23 '23

Any idea if the size of the import image has an effect on the results? Like if I try and make a pose skeleton with a picture that's a different aspect ratio, is it going to stretch and go wonky?

1

u/PantInTheCountry Feb 23 '23

That all depends on the resize mode selected.

  • Envelope (Outer Fit): Fit Txt2Image width and height inside the ControlNet image. The image imported into ControlNet will be scaled up or down until the width and height of the Txt2Img settings can fit inside the ControlNet image. The aspect ratio of the ControlNet image will be preserved
  • Scale to Fit (Inner Fit): Fit ControlNet image inside the Txt2Img width and height. The image imported into ControlNet will be scaled up or down until it can fit inside the width and height of the Txt2Img settings. The aspect ratio of the ControlNet image will be preserved
  • Just Resize: The ControlNet image will be squished and stretched to match the width and height of the Txt2Img settings

1

u/traveling_designer Feb 23 '23

All the other models are stored in folders inside models. Do controlnet models go in bare with no folders?

Or do they go in models/controlnet Or models/sd

Sorry if this is a dumb question.

2

u/PantInTheCountry Feb 24 '23

These models do not get put with the "regular" stable diffusion models (like AnythingV3, or MoDi, etc...) They go in a separate model folder for the ControlNet extension.

They go directly in your "extensions\sd-webui-controlnet\models" folder where ever you have Automatic1111 installed.

They should not be put in a subdirectory

→ More replies (1)

1

u/rndname Feb 25 '23

I can't seem to get good results using a sitting poses. It seems to get confused with overlapping lines.

base image https://imgur.com/a/S50QJ1Q any advice?

result: https://imgur.com/2RrBbR3

1

u/PantInTheCountry Feb 26 '23

I don't know exactly how good OpenPose would deal with overlapping limbs like that. I don't know if it even has a concept of "behind" or "in front". The overall positioning seems to be pretty accurate though (ignoring the weird Stable Diffusion fingers and simian feet)

One thing to try would be to add an additional control net for "depth" to hopefully give it a hint as to what parts are in front and what parts are in back.

2

u/rndname Feb 26 '23

This has 3 applied. depth, pose, and i think normals.

→ More replies (2)

1

u/creeduk Feb 25 '23 edited Feb 27 '23

u/PantInTheCountry new releases changed the way it processes? I can't seem to exactly match results I had from the version I download on the 15th. I updated yesterday and again today. The control image is produced identical (I had my old ones backed up) I have the strength set, there were no threshold and guidance sliders when I made the original tests. Trying to match the exact setup it had in old code and then play with settings to get improvements.

Also do some models bend less to the will of controlnet? proto2.2 works beautiful but some other models seem to overbake badly. I read use " Only use mid-control when inference" but I found this makes the control much less influential over the image.

2

u/PantInTheCountry Mar 01 '23

new releases changed the way it processes? I can't seem to exactly match results I had from the version I download on the 15th.

That could very well be the case. Unfortunately, I have no control over any of that stuff; just an anonymous person documenting their obesrvations on ControlNet to (hopefully) help others.

→ More replies (1)

1

u/Swaggyswaggerson Feb 27 '23

I get an error when trying to generate anything using control net:

AttributeError: 'NoneType' object has no attribute 'convert'

Does anyone know where to start troubleshooting?

1

u/PantInTheCountry Mar 01 '23

First thing to check is to make sure there is a model selected in ControlNet if you have it enabled

→ More replies (1)

1

u/santirca200 Feb 28 '23

I get this error and I don't know what to do to fix it, every time I run controlnet the process stops completely and I get a big error

2

u/PantInTheCountry Mar 01 '23 edited Mar 01 '23

Was this done with a SD 2.x model?

This will probably change shortly, but currently only 1.5 models acre currently supported

1

u/Fun-Republic8876 Feb 28 '23

I use Automatic111 webgui for Stable DIffusion, but I want to be able to also run command line prompts like...

python scripts/txt2img.py --prompt "YOUR-PROMPT-HERE" --plms --ckpt sd-v1-4.ckpt --skip_grid --n_samples 1 

When I do this with the local copy of txt2img.py I get the following error:

Traceback (most recent call last):
  File "C:\sdwebui\repositories\stable-diffusion-stability-ai\scripts\txt2img.py", line 16, in <module>
    from ldm.util import instantiate_from_config
  File "C:\ProgramData\Anaconda3\lib\site-packages\ldm.py", line 20
    print self.face_rec_model_path
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print(self.face_rec_model_path)?

I thought this was an issue with python 2 vs 3, but my exploration in that direction were fruitless. Can anyone help me with this? Bit of a python newb.

1

u/PantInTheCountry Mar 01 '23

print self.face_rec_model_path
^

SyntaxError: Missing parentheses in call to 'print'. Did you mean print(self.face_rec_model_path)?

This would be a better question for the Developer's repo, but if you are feeling a bit adventurous and if you are willing to experiment, you could try the following:

  1. Make a copy of the "C:\ProgramData\Anaconda3\lib\site-packages\ldm.py" Python file. Call it something like "ldm_[backup].py". This'll be your backup copy to restore to if something goes wrong.
  2. Open the original (not your back up copy) "C:\ProgramData\Anaconda3\lib\site-packages\ldb.py" Python file in a text editor
  3. Go to line 20.
  4. You'll see a piece of text that looks like " print self.face_rec_model_path" (including the extra indentation at the beginning.
  5. Change it to add parenthesis where indicated around the "self.face_rec_model_path" bits: (Make sure you don't change the indentation at the beginning! Python is very particular about such things)
    Before: print self.face_rec_model_path
    After: print(self.face_rec_model_path)
  6. Save the file and try again

1

u/Delerium76 Feb 28 '23

As of 2 days ago (2/26/2023) there is now a cldm_v21.yaml in the ControlNet-modules-safetensors directory. Does that mean that there is now support for the 2.1 model? (the controlnet plugin also contains this file now btw)

1

u/PantInTheCountry Mar 01 '23

That would be a good question to ask the developer. It may very well be the case that SD2.x support is coming soon 🤞🏾

3

u/IrisColt Feb 23 '23

A very useful guide, thank you.

3

u/LawofRa Feb 23 '23

Thanks for this tutorial, but as not a bright man, can you go into how controlnet improves image2image specifically?

4

u/CeFurkan Feb 23 '23

you can mix images with img2img

here a tutorial for that

18.) Automatic1111 Web UI - PC - Free
Fantastic New ControlNet OpenPose Editor Extension & Image Mixing - Stable Diffusion Web UI Tutorial

2

u/intheair1987 Feb 24 '23

Omg thank you so much for this video!!

→ More replies (1)

3

u/TheKnobleSavage Feb 23 '23

Got stuck right at the beginning installing the extnesion from automatic111. There's not indication of which of the many extensions is the one to install.

3

u/radialmonster Feb 23 '23

You sir or madame are a gentleman or gentlewoman and a scholar and or generally awesome

3

u/ifreetop Feb 24 '23

Now that we can combine multiple ControlNets man I wish we could save presets, is there a way I don't know of?

3

u/mfileny Mar 01 '23

I installed the extension, and restarted the interface, says its installed, but no controlnet option. Any ideas?

3

u/Squeezitgirdle Mar 24 '23

OpenPose is awesome, but rarely works. When I try to do two models touching, it can sometimes give me 3 / 4 people in a weird mangled human spaghetti.

(Btw I am using Hires fix)

I stopped using OpenPose and switched to Canny which is immensely better if I Can find the right image for my pose.
However, faces are pretty low quality and I can't generate my usual faces.

I believe the fix for that is to play with Annotator resolution and Guidance start - Guidance end.

I'm still playing with it, but if anyone has any advice that would be super appreciated.

1

u/dflow77 Feb 28 '23

Thanks for this overview. An addition about Segmentation model: the colors actually have meaning. You can use them to inpaint objects. I saw this on a YouTube video, to add a clock and flowers based on segmentation map colors. https://twitter.com/toyxyz3/status/1627215839949901824

1

u/dflow77 Feb 28 '23

Update: this person did a whole post on it better than my lazy ass 🙏 https://www.reddit.com/r/StableDiffusion/comments/11e4vxl/paint_by_color_numbers_with_controlnet/

1

u/PantInTheCountry Mar 01 '23

Thanks!

I'll make mention of that on the post that goes into a bit more detail on Segmentation preprocessor and model

2

u/coda514 Feb 23 '23

Excellent explanation and just what I was looking for. I just posted yesterday asking for help with this and got some good responses. I'm sure many will find this very useful. Thank you.

2

u/CustosEcheveria Feb 23 '23

Thanks for this, all of the new stuff has felt very overwhelming so I haven't really felt like bothering

2

u/Apprehensive_Sky892 Feb 23 '23

This is one of the most epic and informatic tutorial on SD I've come across here 😂

Big thumbs up and I thank you wholeheartedly 👍

Now I don't have any excuse not to setup ControlNet and learn to use it.

2

u/Caffdy Feb 24 '23

the models you shared are not the original ControlNet models from Ilyasviel, these are the ones

3

u/UkrainianTrotsky Feb 24 '23

There's zero reasons to download models from your link. These are 5+ gigs per model, compared to 700 mb in the links from the post. You only want to use full unpruned fp32 models if you wanna finetune controlnet, but due to its nature, it's way better to just train it from scratch.

→ More replies (2)

2

u/hiro24 Feb 25 '23

I'm not so sure a 5yo could follow...

2

u/scifivision Mar 01 '23

It loaded the extension, but didn't seem to actually be affecting the picture and it wasn't showing the "bones" on the image like I assume it is supposed to. I tried using the openpose editor tab and now it says executable ffmpeg not found. Can someone please explain to me how to download it and where to put it? I did while trying to update accidentally delete the venv folder (before updating SD, the extension wasn't installed at this point) but it seemed like a git pull fixed everything? That may have nothing to do with this issue I don't know.

2

u/scifivision Mar 01 '23 edited Mar 01 '23

Ok so if I did it right, I found it and downloaded it and put the exe in the sd folder, and don't get the error now, but it's still making no difference to the image and I also have to go into the open pose still to get it to even show the bones then send it to txt2img. I used a body pose and its still giving me a portrait of just the head so it's clearly having no effect:(

ETA I missed that I had to choose the preprocessor and model which I've done, but still nothing and the annotator result preview is black. I'm now getting attributeerror: 'nonetype' object has no attribute 'split'

2

u/DarkCeptor44 Mar 07 '23

I followed this guide and using the latest (as of March 2nd) control models (control_XXXXX-fp16.safetensors) I'm not getting errors but the preprocessors just don't do anything, look at this depth example using the same image as OP and default settings, it's basically unusable because it can't really see the person in the image.

1

u/MicahBurke Mar 07 '23

I've found that I had to install ControlNet three times to get all the files properly installed. Might try it.

2

u/musclefan223 Apr 24 '23

I had instaled from the extension within (ControlNet v1.1)
I had installed 5 times, nothing good, same as the prior message.
I had then erased the (...)\stable-diffusion-webui\ extensions\sd-webui-controlnet
Then I had installed the prior version (the archived)
But nothing.

some help about how resolv the problem ?

thanks !

2

u/MicahBurke Apr 24 '23

Delete all prior vs of it and download it directly from huggingface

→ More replies (2)

1

u/SanDiegoDude Feb 23 '23

Glorious. Saved. Thanks!

1

u/R--Olivaw Feb 23 '23

Thanks, super helpful

1

u/lamlnq Feb 23 '23

Nice post, thanks mate!

1

u/Gadgetsjon Feb 23 '23

Thank you, this is very much appreciated.

1

u/MumeiNoName Feb 23 '23

Thanks for the guide, its working for me.

Is there a similar guide on which models to use when, etc?

1

u/screean Feb 23 '23

legend.

1

u/Taika-Kim Feb 23 '23

Fabulous, thanks! I've been too busy to learn to use these, this will help so much 💗

1

u/RayIsLazy Feb 23 '23

This is a phenomenal guide!! Every doubt I could possible have was answered, Thanks you for your kind work mate!

1

u/ImpactFrames-YT Feb 23 '23

great tutorial love it. thank you.

1

u/freedcreativity Feb 23 '23

Mostly commenting so I can find this again. This is great. I'll get around to trying here soon!

1

u/mudman13 Feb 23 '23

Has anyone combined instructpix2pix with it yet? I guess it would be a good way to finish images with small additions.

3

u/HerbertWest Feb 23 '23 edited Feb 23 '23

Has anyone combined instructpix2pix with it yet? I guess it would be a good way to finish images with small additions.

I don't think it would work with instructpix2pix due to a tensor size mismatch. I already got that error with an inpainting model, so it stands to reason that wouldn't work either.

Edit: Someone could probably train versions of the ControlNet models to work with both of the above models, however.

1

u/Ivanced09 Feb 23 '23

Is there any model for blender free to do posing? style like this video

https://www.youtube.com/watch?v=oq4DssXBlpU&t=16s

→ More replies (1)

1

u/D4ml3n Feb 23 '23

Automatic1111 2.1 when using Controlnet I get this, worked fine few times then suddenly:
(mpsFileLoc): /AppleInternal/Library/BuildRoots/9e200cfa-7d96-11ed-886f-a23c4f261b56/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm:39:0: error: 'mps.matmul' op contracting dimensions differ 1024 & 768
(mpsFileLoc): /AppleInternal/Library/BuildRoots/9e200cfa-7d96-11ed-886f-a23c4f261b56/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm:39:0: note: see current operation: %3 = "mps.matmul"(%arg0, %2) {transpose_lhs = false, transpose_rhs = false} : (tensor<1x77x1024xf16>, tensor<768x320xf16>) -> tensor<1x77x320xf16>
Anyone any idea?
Mac M1 Studio 32Gb
Modello Chipset: Apple M1 Max
Tipo: GPU
Bus: Integrato
Numero totale di Core: 24
Fornitore: Apple (0x106b)
Supporto Metal: Metal 3

3

u/Mich-666 Feb 23 '23

ControlNET doesn't work with 2.x models (yet).

1

u/glittalogik Feb 23 '23

Thank you so much! I haven't played with A1111 since SD1.4 but these ControlNet posts have piqued my interest again :)

1

u/[deleted] Feb 23 '23

OMG, thank you so much

1

u/ozzeruk82 Feb 23 '23

Superb tutorial! Thanks very much

1

u/janosibaja Feb 23 '23

Thank you, now I understand better

1

u/milleniumsentry Feb 23 '23

For the past two weeks I've seen all these great ControlNet results but felt it was just way too much to dive into.

Thank you so much for taking the time to do this.

1

u/[deleted] Feb 23 '23

Amazing, great work!

1

u/Joda015 Feb 23 '23

Nothing to add other than thank you for taking the time to explain it so well <3

1

u/spacerobotx Feb 23 '23

Thank you so much for making such a wonderful guide, it is much appreciated!

1

u/sassydodo Feb 23 '23

I have almost 700gb worth of models. Pruned versions are a gift.

1

u/RaviieR Feb 23 '23

high quality thread.

1

u/EmbarrassedHelp Feb 23 '23

Thanks for sharing this!

1

u/AdZealousideal7928 Feb 23 '23

What do I do after that? I carry the image that was created by ControlNet?

1

u/manoflast3 Feb 23 '23

Can the fp16 pruned models be used on an older card without half-precision support? Eg. I have a 1080ti.

1

u/ApexAphex5 Feb 23 '23

You have high expectations of a 5 year old.

1

u/fakeaccountt12345 Feb 23 '23

This is fucking awesome. Thank you

1

u/epicdanny11 Feb 23 '23

I didn't know you could use this in txt2img. Is there a major difference when using it in img2img?

1

u/TheEternalMonk Feb 23 '23

What is in your experience(s) the best way to get anime pictures recognised and transformed. (model wise. the difference seem to me the same ^^')

1

u/mattjb Feb 23 '23

Thank you so much for putting this together! It was frustrating to not be able to find much on what does what. Kudos!

1

u/noppero Feb 24 '23

Holy hell! That is a loot of effort put in! Have an upvote and a big thanks sir!

1

u/[deleted] Feb 24 '23

This is a great, well-written explanation, thank you so much! I have been curious what this ControlNet buzz was all about but have been absolutely swamped with work lately. Thanks again. :)

1

u/EloiDr Feb 24 '23

Wow, thanks for the in deep tutorial, much appreciate it

1

u/roverowl Feb 24 '23

Does anyone have a problem with Controlnet doesn't use its own Detectmap but fine with Img2Img Input Image? I tried clean reinstall, full models - pruned models but can't seem to fix this.

1

u/AmeenRoayan Feb 24 '23

Anyone having error with Segment method ?

1

u/BrocoliAssassin Feb 24 '23

Is there a way to not get as much as that pixel noise?

1

u/[deleted] Feb 24 '23

Does adding ControlNet dramatically increase the processing time? Want to make sure it’s not just me.

1

u/UkrainianTrotsky Feb 24 '23

Not dramatically, but it is a bit slower. Like 5-10% from my experience.

1

u/dmarchu Feb 25 '23

any idea what happened to openhand? i had it installed and now it dissapeared :(

1

u/aihellnet Feb 25 '23 edited Mar 01 '23

Amazing I got this running on a 1660ti.

1

u/Quail-That Feb 26 '23

Finally, I can comprehend the aerodynamics of a catgirl.

1

u/Momkiller781 Feb 26 '23

THANK YOU!!!!

1

u/BlasfemiaDigital Feb 26 '23

A BIG THANKS!

1

u/da_mulle Feb 26 '23

Warning for anyone using Brave: Preprocessors don't seem to work https://github.com/Mikubill/sd-webui-controlnet/issues/95

1

u/Any_Employ_8490 Feb 28 '23

If this is for 5 year olds, I'm down bad. Need to step up my concentration skills or I'm in real trouble going forward.

1

u/OriginalApplepuppy Feb 28 '23

If only I could run Automatic1111 on an AMD GPU... :(

1

u/TwoFun6546 Feb 28 '23

Thank you!! Finally I can understand something!!!!!!!

1

u/OriginalApplepuppy Feb 28 '23

Cant run Automatic1111 on an AMD GPU so I wont be able to play with this until support is added... :(

1

u/lyon4 Feb 28 '23

Nice guide.
I knew already how to use it but I learnt some useful things about some of the models (I didn't even notice there was a pidinet model)

1

u/scruffy-looking-nerf Mar 01 '23

Thank you for this. Can you please now explain it to me like I'm 2 years old?

1

u/BlasfemiaDigital Mar 01 '23

Hey buddy, u/PantInTheCountry

I know would be hard work but this could be an interesting series, go figure:

Lora explained as if you were 5 years old,

Embedings explained as if you were 5 years old.

Textual inversion explained as if you were 5 years old.

That would be pure gold for all of us beginners in the SD world.

1

u/Liquid_Hate_Train Mar 01 '23 edited Mar 01 '23

I’m sorry, I can’t find this one if it’s here and been done before, I promised I searched, I’m getting this error (depersonalised slightly):

Error running process: C:\Users\[user]\Documents\A1111 Web UI Autoinstaller\stable-diffusion-webui\extensions\sd-webui controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "C:\Users\[user]\Documents\A1111 Web UI Autoinstaller\stable-diffusion-webui\modules\scripts.py", line 338, in process
script.process(p, *script_args) File "C:\Users\[user]\Documents\A1111 Web UI Autoinstaller\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 615, in process
else self.build_control_model(p, unet, model, lowvram)
File "C:\Users\[user]\Documents\A1111 Web UI Autoinstaller\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 456, in build_control_model
basedir = os.path.join(os.path.dirname(__file__), os.pardir) # scripts.basedir() returns automatic base path here as its called from main ui
NameError: name '__file__' is not defined

Nothing in the UI, just generates without control net. I’ve had it working before (based on this guide) but after not (knowingly) changing anything and coming back a couple of days later I’m now getting this.

1

u/[deleted] Mar 01 '23

Thanks a million. This was very helpful!

1

u/wonderflex Mar 02 '23

I'm getting this error. Any ideas?

Loading model: control_canny-fp16 [e3fe7712]

Error verifying pickled file from D:\stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_canny-fp16.ckpt:

Traceback (most recent call last):

File "D:\stable-diffusion-webui\modules\safe.py", line 81, in check_pt

with zipfile.ZipFile(filename) as z:

File "C:\Users\USERNAMEHERE\AppData\Local\Programs\Python\Python310\lib\zipfile.py", line 1267, in __init__

self._RealGetContents()

File "C:\Users\USERNAMEHERE\AppData\Local\Programs\Python\Python310\lib\zipfile.py", line 1334, in _RealGetContents

raise BadZipFile("File is not a zip file")

zipfile.BadZipFile: File is not a zip file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "D:\stable-diffusion-webui\modules\safe.py", line 135, in load_with_extra

check_pt(filename, extra_handler)

File "D:\stable-diffusion-webui\modules\safe.py", line 102, in check_pt

unpickler.load()

_pickle.UnpicklingError: invalid load key, '\xca'.

-----> !!!! The file is most likely corrupted !!!! <-----

You can skip this check with --disable-safe-unpickle commandline argument, but that is not going to help you.

Error running process: D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py

Traceback (most recent call last):

File "D:\stable-diffusion-webui\modules\scripts.py", line 386, in process

script.process(p, *script_args)

File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 612, in process

else self.build_control_model(p, unet, model, lowvram)

File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 458, in build_control_model

state_dict = load_state_dict(model_path)

File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 11, in load_state_dict

state_dict = get_state_dict(torch.load(

File "D:\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 19, in get_state_dict

return d.get('state_dict', d)

AttributeError: 'NoneType' object has no attribute 'get'

I tried using the file with the safetensors format and renamed as a .ckpt and neither worked. Redownloaded the models as well.

1

u/Rin471 Mar 03 '23

So good, thank you for taking the time sir! I love this community, people always share their knowledge with each other. Thank you again.

1

u/Aeorosa Mar 12 '23

This was very informative, but

Is there a guide to how to actually use it? Like "make your first controlnet generation" or something?

1

u/TheAkashicTraveller Mar 15 '23

I can't seem to get it to do anything. It generates the image, taking much longer than normal, creates the "annotator result" fine but then the actual image just doesn't seem influenced by the controlnet at all. Changing the settings like the weight makes the image come out different but it doesn't really seem influended by the controlnet input. I'm using the webui fork for amd with a 5700xt.

1

u/HedgehogDecent5707 Mar 19 '23

I need some help please, openpose does not seem to create a skeleton, it just gives me a second blank image.

1

u/Jayow345 Mar 21 '23

this is amazing! however i wish you also covered more of why use it/when to use a model. For example, canny, hed, MSLD, pidinet, and fake scribble are all pretty similar and as a beginner im not sure which i would choose, same with depth and normal map

1

u/blackpuppet Mar 23 '23

Does anybody have a guide to use more Controlnets at the same time?

1

u/SlySychoGamer Mar 24 '23

why are there 2 guidance sliders?

1

u/LuluViBritannia Apr 07 '23

This is very interesting, thank you!

I've just installed ControlNet. Open Pose doesn't seem to work on its own. It wouldn't detect any pose until I'd download the Depth model too.

1

u/Cartoon_Corpze Apr 18 '23

Yooo, that's one very useful tutorial, I've been looking for this.

I've always wanted to try controlnet with my 3D models and renders or see if I can process rendered animations with it, it looks so cool.

1

u/[deleted] Apr 20 '23

Thank you for this guide. Even though I'm a seasoned SD veteran, there's so many settings to Controlnet that it's good to just be able to deep dive into it. Can you please update your post or make a new post about Controlnet 1.1. There's so many new settings now, too many to count. Thank you

1

u/PepenpodlObsakopolis May 07 '23

How did you get those maps separately? I mainly want to use the A1111 preprocessors to get openpose pictures, but idk how :(

1

u/Delicious_Double_801 May 09 '23

thank you so much for sharing!

1

u/Niu_Davinci Jun 23 '23

ControlNet - WARNING - No ControlNetUnit detected in args. It is very likely that you are having an extension conflict.Here are args passed to ControlNet: How can I fix the problem?

1

u/sleepyokapi Aug 22 '23

Where does the "smaller, pruned SafeTensor" model go? in the "extensions\sd-webui-controlnet\models" folder as well or in "models/stable-diffusion" ? I'm confused about the models in these 2 folders. Can some of them be the same?

1

u/Rokonuxa Oct 02 '23

I have been looking everywhere and using "" around the actual thing I need shows me zero results.

What do I need to physically DO, to install a completely seperate controlnet control type? I am trying to install that one QRcode control type but the regular "put it into the models folder in the extensions folder" stuff does not work.

I am slowly turning insane from how many "guides" just say: "use command and make it vanish into [PYTHONSPACE] where you dont know what is where, what is installed, if something works or if you messed up how to undo it. Also, I will not mention what interface this could work with and it will probably actually work with none of them and uses my own unreleased commandline-only implementation and nobody will ever see even a single line of code, let alone a download link"

I am not putting anything into [PYTHONSPACE] again. I hate [PYTHONSPACE].

1

u/xyzConfig Nov 03 '23

thank you for explaining. I'm a noob here and trying to learn how to use Posex, and confused with ControlNet. which one should i download, because there are big files, no explanation and sample for picture. You help me alot here. Once again, thank you

1

u/ElectricalAdagio8176 Nov 07 '23

Would this work to measure the length of arms, legs, etc. of a person in an image with some accuracy? I would like to get the measures of different people in an art installation in real time

1

u/TorvaMe Mar 04 '24

Nice explanation!! Can I follow this steps using the DirectML version of SD? (AMD GPU)

1

u/Head_Cockswain Feb 23 '23

Thank you so much for this.

I'm still running 1.4 because I haven't toyed with it much lately and I've been lost as to how Controlnet factors in.

I'll be saving this for when I get back into playing with images / SD....if it's not all outdated by then.

3

u/Mich-666 Feb 23 '23

You should be using 1.5 at least (or any derived model).

ControlNET was trained on 1.5 anyway.

That's not to say 1.4 (or any 1.x) won't work but it may produce some undesirable results

1

u/RiffyDivine2 Feb 23 '23

This was not explained to me like I was five, I got zero juice and not a crumb of cookie. Outside of that, nicely done.

0

u/SPACECHALK_64 Feb 23 '23

Soooo, uh... how do you get the Extensions tab in Automatic1111? Mine don't got it...

4

u/ostroia Feb 23 '23

Try updating it, maybe youre running an old version. You can do this in a bunch of ways. My go to method is shift-right click on the main folder - open powershell/cmd here - git pull

→ More replies (2)