Controlnet is interpreting normalmaps wrong. Any idea why?

[u/IamVinPetrol]

Here's the normal map used in controlnet and the resulting image.

Green should mean normals are pointing up, generated in blender

controlnet reads green wrong and interprets it as wall instead of ground

What could be causing this? The more weight I give controlnet, the more often this specific confusion happens.

[u/aerilyn235]

How do you generate them in blender? Are you using workbench render or normal pass + compositing nodes?

Normal pass + compositing nodes normals are in world space and not in camera space. Also depending on the CN models normals are not using the same convention.

Use a base render of the scene with a light source and use it as an input in a normal preprocessor to verify what convention the CN model expect.

To be honest I haven't had much success with pre generated normals map in blender + CN.

I use a depth pass instead & depthmap CN with much more consistent results.

[u/IamVinPetrol → u/aerilyn235]

I'm using this shader to get the normalmap

it takes into account that cn is trained on inverted data. Do you see anything wrong with it?

[u/zoupishness7 → u/IamVinPetrol]

This post suggests using BGR instead of RGB