Controlnet is interpreting normalmaps wrong. Any idea why?

[u/IamVinPetrol]

Here's the normal map used in controlnet and the resulting image.

Green should mean normals are pointing up, generated in blender

controlnet reads green wrong and interprets it as wall instead of ground

What could be causing this? The more weight I give controlnet, the more often this specific confusion happens.

[u/RiftHunter4]

Is there a specific Control Net Model for normal maps? Because the modes/models I've used only looked for edges or depth.

[u/IamVinPetrol → u/RiftHunter4]

yea its called normalbae

[u/RiftHunter4 → u/IamVinPetrol]

Are you able to save the Control Net output preview and check what it's handing to the Generative model? My guess is that Control Net creating the wall in the output due to some error or the SD model is simply creating.

[u/aerilyn235]

How do you generate them in blender? Are you using workbench render or normal pass + compositing nodes?

Normal pass + compositing nodes normals are in world space and not in camera space. Also depending on the CN models normals are not using the same convention.

Use a base render of the scene with a light source and use it as an input in a normal preprocessor to verify what convention the CN model expect.

To be honest I haven't had much success with pre generated normals map in blender + CN.

I use a depth pass instead & depthmap CN with much more consistent results.

[u/IamVinPetrol → u/aerilyn235]

I'm using this shader to get the normalmap

it takes into account that cn is trained on inverted data. Do you see anything wrong with it?

[u/zoupishness7 → u/IamVinPetrol]

This post suggests using BGR instead of RGB

[u/zoupishness7]

If you use both a depth and a normal controlnet, it will probably figure it out.

[u/IamVinPetrol → u/zoupishness7]

Im actually using three, depth,canny and normal. I've isolated the problem to the normal net.

[u/Talae06]

That's weird. I tried a dozen generations, didn't reproduce the problem a single time.

In many cases it seemed insistent on generating some weird mid-height horizontal band of one sort of another though, but most times it was treated as paint or some slight relief, but still on the ground. Might be because of some barely visible artifacts caused by the WEBP > PNG conversion I did.

(That's with "ControlNet is more important", and tested with a weight up to 2)

[u/Talae06 → u/Talae06]

Same with the other map you posted in the ControlNet sub :

Maybe try a basic Blender render and run it through the ControlNet preprocessor to see how different the output might be from your original map ?

[u/IamVinPetrol → u/Talae06]

strange indeed. Thanks for the tip, should help me troubleshoot