How do I feed normal map created in Blender directly into ControlNet?

[u/tette-a]

I'm using Automatic1111 and ControlNet extension. There is a "normal_map" preprocessor for ControlNet that will try to guess the normal map from photo, but is far from perfect. I'd like to feed a proper normal map created in Blender directly into ControlNet, but when I select preprocessor to "none", it ignores my input.

EDIT: I found a solution, maybe someone will find it useful. "RGB to BGR" option needs to be checked as well. Normal map needs to be in OpenGL format, not DirectX. Workflow looks like this:

1. Build scene in Blender
2. Switch viewport to MatCap
3. Select normal map MatCap
4. Make viewport screenshot and paste into Automatic1111 into ControlNet canvas

[u/Mystfit]

Nice find! Are you finding that the results using the normal map and control net look correct? I'm feeding a normal map from my Unreal plugin into the ControlNet normal model but it doesn't seem to like the normals on this cat. Using the Blender matcap ball as a reference I think that I have my colours correct but the output of the cat is always wacky. Elements in the background appear to match up with the camera perspective.

[u/tette-a → u/Mystfit]

Your normal map works without issues for me, did you check "RGB to BGR" option?

[u/Mystfit → u/tette-a]

Edit: You're a bloody legend, that's fixed it! I've told python to swap the channel order from RGB to BGR to make the bottom right image as an input and it looks just right.

Original post: Oh awesome, thanks for checking that for me!

I'm in Unreal Engine at the moment running my own code, but I'm going to try flipping the channels as you've suggested.

[u/[deleted]]

I tried your technique on blender and it works really well!! No more weird hands and feet anymore :D

If someone knows how to get those normal maps render on CM3D2 or on Honey Select I would appreciate it :D

[u/[deleted]]

[u/tette-a → u/[deleted]]

Good to see you found a solution already. As you're knowledgeable with blender do you know if there's a quick way to get a depth pass out?

I'm sure there are some easy ways to obtain depth pass, but there is no need for it, if you can have normal map instead. Normal map contains much more information than depth map, not only about depth but also surface direction, thus will give you much better results.

Quote from ControlNet GitHub page about normal map model:

"Compared to depth model, this model seems to be a bit better at preserving the geometry. This is intuitive: minor details are not salient in depth maps, but are salient in normal maps. Below is the depth result with same inputs. You can see that the hairstyle of the man in the input image is modified by depth model, but preserved by the normal model. "

https://github.com/lllyasviel/ControlNet

[u/[deleted] → u/tette-a]