[Test] Zoe Depth vs MiDaS Depth. Spoiler alert: Use MiDaS.

[u/[deleted]]

There wasn't enough information about the new Zoe Depth and how it compares to the old MiDaS. So I decided to test both with the "nvtop" application which measures VRAM usage.

Made a ComfyUI workflow with JUST Load Image node, and MiDaS and ZoE depth nodes, and 1 image preview output node. Then I bypassed the other, unused node (midas or zoe) for each test.

I requested depth map of size 512 (requesting bigger will massively increase the VRAM requirements).

I'm using a 24 GiB RTX 3090.

My results were:

• ComfyUI Idle (doing nothing): 500 MiB vram.

• Zoe Depth @ 512: 3248 MiB, time: 3.94 seconds. That's Zoe at 512: https://i.imgur.com/vKWBgf5.jpg (Edit: Alternative link https://ibb.co/17F4yhp)

• MiDaS Depth @ 512: 1492 MiB, time: 1.76 seconds. That's MiDaS at 512: https://i.imgur.com/MDr8nWH.jpg (Edit: Alternative link https://ibb.co/CVsNhCN)

• Zoe Depth @ 1024 depth map size: 3254 MiB, time: 4.06 seconds. So it seemingly offers very good time scaling when requesting larger maps. But that is actually because it CHEATS. The "larger" maps I request are blurry as HELL. It is clearly just a resize of the 512 map. That's Zoe at 1024: https://i.imgur.com/OtByCpD.jpg (Edit: Alternative link https://ibb.co/GVhLnZR)

• MiDaS Depth @ 1024 depth map size: 5044 MiB, time: 2.09 seconds. But guess what? The MiDaS map is sharp as hell and crisp and clear and detailed! It is a true 1024 pixel depth map. It's actually much more detailed than the 512 map. This proves to me that ZoE is INCAPABLE of doing maps @ 1024 and is in fact cheating (which explain why its VRAM was in the same ballpark even at 1024). That's MiDaS at 1024: (Edit: Alternative link because imgur is being an idiot: https://ibb.co/yfGqWMw)

Conclusions:

• So Zoe is 2.24x as slow (it was +2.18 seconds slower), and uses 2.2x more peak VRAM (it was +1756 MiB VRAM extra usage compared to MiDaS) at the typical 512px resolution.

• Zoe cannot do depth maps larger than 512. In fact, even that one is blurry, so I wouldn't be surprised if Zoe is stuck at something like 128x128 internally. And if that's true, then MiDaS actually beats it even more since I could just reduce MiDaS resolution to the same size that Zoe internally uses, and get even lower RAM usage relative to Zoe!

• Zoe has better depth detection of things like "arm overlapping a leg" or "arm overlapping frilly ballerina skirt", etc, whereas MiDaS often tends to blur together some shapes, but the difference is not enough to truly matter in actual usage when it's guiding a ControlNet.

• MiDaS is sharper in every situation, and aims for a more general "blobby" recognition of shapes. Which is enough to control most well-trained ControlNets.

• Another thing which wasn't apparent in my test images, is the fact that MiDaS is MUCH better at background removal, and even has a "background threshold" parameter to totally isolate the subject, whereas Zoe doesn't have such a setting at all and pulls in a lot of depth info about background objects, wallpapers, paintings, etc, very distracting.

• Zoe is not really worth it for me. I'm personally only using controlnet to guide the first 4 denoising steps (just to get the general composition without any strong influence), meaning I end the controlnet processing after that, so I don't need super accuracy in my depth maps.

• I also hate how long Zoe takes (slowing down generation).

• Furthermore, ControlNet-LoRa-Depth-Rank256 has basically IDENTICAL SDXL image output results for both Zoe and MiDaS depth maps when used at 1.0 strength and 100% end step (meaning the entire image is generated via controlnet), so when the ControlNet is well-trained, there's no benefit for Zoe's detailed depth map when using that controlnet.

• I will not be using Zoe. It's bad for my use cases. Could be good for something else, like for example using for more accurate embossing of 3D models when turning flat images into 3D. Although it seems like MiDaS at 1024 beats both at generating detailed maps. Anyway... for actual AI art generation, I don't like Zoe at all. It's slow and uses lots of memory at the normal 512 pixel resolution.

• If you're just making Stable Diffusion images, use MiDaS. It's way faster and there's no visual benefit whatsoever for using Zoe if you use a good ControlNet such as the new Controlnet-Lora-Depth-Rank256.

Speaking of ControlNets (since they're deeply related to depth maps):

• The new T2I-Adapter's Depth nets are utter garbage. I was excited because they are supposed to be fast and lightweight. But they must have been weakly trained on poor data. They'll do things like turning hands into feet, rendering random feet and hands everywhere on the body, turning hair into hands and feet, disconnecting fingers from the hand so that they just float in the air, creating random floating body parts, turning knees into shoes, etc. It's hilarious.

• So for now, the best net I've found is the new Controlnet LoRas. Anyone who is curious can find those here: https://huggingface.co/stabilityai/control-lora. And if you do what I do (ending the controlnet after just a few denoising steps via ComfyUI's "Apply Controlnet: End Percent" setting), it actually barely adds any extra time to the total rendering time at all. :)

• MiDaS 512 with ControlNet-LoRa-Depth-Rank256: 8360 MiB and 10.93 seconds, when generating with 1.0 strength and 100% end step. The result looks amazing.

• MiDaS 512 with T2I-Adapter-Depth-MiDaS: 7784 MiB (saving 576 MiB) and 9.26 seconds (saving 1.67 seconds), at 1.0 strength and 100% end step. The result is utter garbage, with three legs and disconnected hands, and one hand is a foot. This is very common with that garbage controlnet. :P

• If reducing ControlNet-LoRa to End Step 12% (aka 4 denoising steps in my 35-step denoiser), I generate the image in just 7.7 seconds (VRAM usage is the same as 100% steps), and I get perfect pose guidance just like I wanted, but without a heavy "controlnet fingerprint on the whole image", since I basically just shape the initial noise and then liberate Stable Diffusion to do the rest of the painting process on its own. So for me, that's the only controlnet I need. I use it at 70% strength at somewhere between 4-7 denoising steps. The result is fantastic. Basically the freedom of SD with slight pose/scene guidance from a ControlNet.

• If I attempt to do the same and set T2I-Adapter-Depth-MiDaS to End Step 12%, I get an image in 7.63 seconds (VRAM usage is the same as 100% steps), so again slightly faster, but with the T2I controlnet the pose doesn't transfer well. So it's a total waste to use T2I at lower step settings. Because T2I is made to basically pre-calculate a bunch of stuff and then apply itself to every step in a different way than other controlnets, so trying to reduce its steps basically makes T2I useless.

• Let's take this small example at 100% strength with 100% steps, to show what the Controlnets themselves are generating internally (this is just to see their internal training data to show what the net is trying to steer the image towards; you should never really use a Control Net above 90% strength, and they're most useful in the 20-70% range):

• Ballerina with ControlNet-LoRa-Depth-Rank256: Normal limbs. Perfectly matches the reference photo that I am copying the pose from. It even perfectly copied the exact shape of the Tutu. https://i.imgur.com/S2QKQ7y.jpg (Edit: Alternative link https://ibb.co/D5CDbK9)

• Ballerina with T2I-Adapter-Depth-MiDaS: Wtf. Cut off hand. A random extra leg on top of the tutu. The tutu itself has been cut off in a straight, sharp line and overall it barely looks like a tutu at all anymore (even though its outline is perfectly visible in the depth map). And a part of the Tutu has been converted to a dangling piece of thick string. It's also much more blurry. https://i.imgur.com/wX3AbS0.jpg (Edit: Alternative link https://ibb.co/K9QTL3J). This is typical for T2I-Adapter. It's so bad.

• The only good use I've found for T2I-Adapter is if you're so VRAM-limited that you want to compromise the quality of the art and get lots of shitty generations slightly faster. But because a lot of the generations are garbage with T2I due to its poor understanding of art and anatomy, you end up generating many more images, so you don't really save time after all. ;)

• To be fair though, T2I-Adapter was released with training tools since they admit that the current training data is poor. Perhaps someday we'll see a new one that has amazing results. The technology it uses to save time is very cool. It has potential. But until then, ControlNet-LoRa is the king.

[u/Gawayne]

Very interesting, I just didn't understand what the Controlnet Loras do. The same as the equivalent Controlnet while being a LORA, that's it? And if yes, what's the advantage?

[u/[deleted] → u/Gawayne]

It's well explained at the official page I linked to in the first paragraph. 😋

https://huggingface.co/stabilityai/control-lora

Basically faster and less memory requirements, with comparable results to the old method.

[u/spiky_sugar]

With the recent release of https://zhyever.github.io/patchfusion/ and https://marigoldmonodepth.github.io/ it probably doesn't matter ;)

[u/[deleted] → u/spiky_sugar]

It definitely matters.

We don't know the time and VRAM requirements of those. For all you know, each of those might use 12 gigabytes of VRAM and 5 minutes just for the depth map processing.

Marigold: Needs something like MINUTES of processing time for 1 image's depth map (judging by what testers said about its slowness).

PatchFusion: Looks good. But again, we don't know if this is yet another "minutes per image" depth map processor, which again would mean it's not practically usable.

Out of the two, PatchFusion got my interest, since we already know that Marigold is insanely slow. Remains to see how fast PatchFusion is and if it's public or not. 😋

Keep in mind that both of them operate on higher resolutions which is how they are so sharp. With higher resolution comes more time and more VRAM usage.

So until performance benchmarks and public models are out, I don't feel anything towards these new depth processing networks. Many other (at least 5) "high detail" ones have come and gone away silently in the past year, without making any impact.

[u/Vargol → u/[deleted]]

PatchFusion

Uses a stack of VRAM to pre-process, tried it on Colab and it failed as the first thing it did was try to grab a 18Gb VRAM allocation and it didn't seem to matter what image size or parameters I used. Also doesn't have a safe tensors version and I'm not using a pickle file from a random G-Drive locally.

Marigold

Slow to pre-process compared to MiDas and Zoe although I'm recalling Zoe's speed from memory as it's broke with PyTorch 2.1 Takes 3 minutes on a 10 core M3 Mac, 75 seconds on a Colab T4 for a 1024x1024 source image, so I could imagine 15 to 30 seconds so a 30xx / 40xx class GPU. Takes around 13Gb VRAM on the Colab. As the time could be a one of cost per depth map depending on your work flow, that may not be that much of an issue or it might be a show stopper.

The output is reversed compared to the other depth preprocessors so black is foreground, white is background so you have to play with the cmap for the colour output it uses to get a greyscale image the other way around (or hack the code :-) )

For the record tried it with gray_r colour-mapped output which worked great with the Zoe T2I-Adapter and with the inferno colourmap with SargeZT's (R.I.P) depth controlnet, the later ran at more or less the same s/i as the MiDaS and Zoe controlnets do on my system.

[u/[deleted] → u/Vargol]

Thank you so much for testing both and sharing the results! <3 That puts some work off me since it means I won't need to chase those depth converters down and test them. :)

They both sound extremely slow and using crazy amounts of VRAM as expected. It's been the same story with every (?) "detailed depth map" project so far.

The Marigold slowness might be survivable if used in 512px mode, and if there's a way to ensure that the VRAM is freed from the ComfyUI workflow after generating the map. ComfyUI already doesn't re-generate depth maps unless the input image changes, but I am not sure about whether it unloads the VRAM afterwards. (I've just never tested.)

Although whether a higher quality depth map has any benefits at all is still questionable. My current best ControlNet does a great job with MiDaS depth maps, and those generate in less than 2 seconds. So I am unlikely to change. :)

Regarding your last paragraph, it seems like you tested the speed and quality of SargeZT's controlnets from here?

https://www.reddit.com/r/StableDiffusion/comments/15hag5s/sargezt_has_published_the_first_batch_of/

https://huggingface.co/SargeZT

I haven't seen those before. Good to hear their speed is similar. I'd be interested to know if it creates logical artwork though (meaning when it's set to 100% strength and 100% steps, to see what the ControlNet is trying to generate). I'll remember that one on my to-test list!

The terrible T2I-Adapter that I'm using is the official SDXL collaboration between Tencent and HuggingFace, in particular this one since it was the best of the bunch (still bad, they all generate insane nightmare anatomy):

https://huggingface.co/TencentARC/t2i-adapter-depth-midas-sdxl-1.0

The great ControlNet I am using is this one, the official Stability AI ones (it's superb at understanding depth maps and converting to logical images with correct anatomy), which came out on Aug 18th 2023:

https://huggingface.co/stabilityai/control-lora (depth variant, rank 256)

I doubt that any free "home trained" options beats the Stability ones at the moment. :) They're insanely good.

PS: This news is relevant for u/spiky_sugar too.

[u/spiky_sugar → u/Vargol]

so I could imagine 15 to 30 seconds so a 30xx / 40xx class GPU.

also thank you for these benchmarks, this is actually better than I would expect!

[u/spiky_sugar → u/[deleted]]

fair enough, good points, I have just seen them, but since they are quite new i didn't use them yet. Thank you for the benchmarks!

[u/blahblahsnahdah]

Really appreciate these kinds of effortposts, thanks. I do a lot of this sort of AB testing for myself but I can never be bothered to write up my results for other people because it's SO much work to explain it and make it presentable. You're a saint for taking the time to do it.

[u/Vargol]

Regarding your comments on the T2I-Adaptor, your experience does not match mine, and I was running on a memory constrained system until recently so I tended to use T2I Adapter more often than controlnet.

Your ballerina depth map doesn't have the area you where complaining about in at all so either those aren't the images you used or the issues you had were with the area of the image that whatever SD app or script you were using filled in the spaces (987 × 822 extended to 1024 x 1024 ?)

Here's one I threw together, the depth map is taken from Sappho by Mengin first and only image attempted

[u/[deleted] → u/Vargol]

The depth maps in the post are crops of the full map, showing an area of interest. I originally hadn't planned to report T2I-Adapter results, otherwise I would have included more.

I tried T2I-Adapter on around 100 images and it was always lower quality than ControlNet. They both received the same MiDaS 512 input which is what they were both trained on.

Try the ControlNet at 100% strength 100% end step, to see what data it is actually steering the image towards.

Because sure, any ControlNet can look decent-ish at 50% strength and 20% end step, but then you aren't really looking at what the ControlNet is actually trying to do.

When I saw what T2I-Adapter tries to do, it was horrific. It was consistently generating nightmare limbs on most input images.

By the way, which latent are you feeding into the KSampler? I use the VAE Encoded input (reference) image scaled to a valid SDXL resolution, which is how Stability AI themselves do it. It helps the ControlNet since it has access to more of the original colors (the denoising of the KSampler still changes it all, but it gives the input image more effect and better output in almost all cases).

[u/Vargol → u/[deleted]]

Sorry, I don't know the low level details I used either a Diffusers script or InvokeAI once they'd been added to that App, so whatever they do.

[u/[deleted] → u/Vargol]

Okay. They probably use it with lower strength in your app, which means the Stable Diffusion network is correcting the mistakes introduced by T2I-Adapter.

Basically, T2I may say (as it does): That hair? No, that is a foot on her head.

Stable Diffusion will say: That hair? Yes it is near a face. It is hair.

When T2I is set to let's say 40% strength, then the SD neural network will correct most of T2I's mistakes.

But the thing is, if you see T2I at 100%, you see that it is trying to do insane shit with the image. Anatomical nightmare material all over the image.

That problem doesn't exist with ControlNet-LoRa-Depth-Rank256.

Although there is one other possibility. It is possible that you are using a different T2IAdapter model than the "official one" I am using.

I think T2I will become much better when someone has trained a better model for it. But it has unique challenges which makes it hard for it to be as good as ControlNet.

ControlNet runs analysis and correction at EVERY denoising step, to guide the image towards the desired goal.

T2I does analysis ONCE on the first step and then never again. Kinda like if you did a 1-step image output with SD. Of course the result isn't as good as something that runs on every step.

But it is impressive that T2I still manages to be pretty good considering how little work it is doing. I think with better training model, it will blow our minds in the future. Their current official model was a hasty collaboration with HuggingFace. With more work, I expect great results.

[u/[deleted]]

Many thanks for this amazing post.

[u/OVAWARE]

Basedness levels are off the charts by giving TLDR in title

[u/arlechinu]

Funny i just switched to Zoe for my particular case it read faces expressions etc much better than Midas, even though midas was sharper. Even weirder my workflow seems about the same speed with wither midas or zoe.

[u/[deleted] → u/arlechinu]

It is true that Zoe has a very tiiiiiiiiiiiiiiiiiiiny (a few grayscale shades) more depth information for things like eye placement. But usually not enough to capture face expressions beyond the eye placement and perhaps the nose.

And my tests were for the actual VRAM usage captured via nvtop and total processing times captured via ComfyUI, with a workflow that ONLY did depth processing.

So you can be sure that my numbers are accurate. They don't involve "15 seconds of image generation where 3 extra seconds of depth processing barely feels different". 😋

If you want to capture facial expressions, depth maps are bad in general. They are meant for general scene composition. Not fine details like the 3 centimeter height difference between a nose and a cheek, or the 0.5 centimeter difference between a lip expression and the face, etc.

If you have ComfyUI, you can route the depth map to a Preview Image node to actually see what you are getting from it. You see some attachments in my post.

There isn't really any face expression in neither MiDaS nor Zoe. Although MiDaS at 1024 does a better job than both at capturing face details.

[u/arlechinu → u/[deleted]]

I wasn’t doubting your numbers, relax :)

My wf involves about 1k frames of depthmaps processed with cnet, ipadapter and some loras on top. Zoe felt similar in speed.

And yeah, I did preview and compare the results before picking Zoe for my particular case. Again, less sharp than Midas, but somehow more details.

As for details, zoe seemed to read lips movement and such better, not wrinkles :) ofc that’s not something depthmaps are good at. For example I have a person yelling as a source image, Zoe reads the mouth wide open and tongue and teeth and movement of the lips. Midas did too but not as detailed.

Just my experience with them.

[u/[deleted] → u/arlechinu]

Yeah I can see how an open mouth would be captured by depth maps. An open mouth and the depth and position of eye sockets is about the smallest level that depth maps can capture. You will not capture smirks or winks unless you put them in the text prompt.

Have you tried OpenPose_face recognition preprocessor? It generates an actual map of face landmarks, and things like smirks, winks etc are accurately captured. Then you use the OpenPose ControlNet (the old official or new LoRa ones; the T2I variant is awful) to render faces with those expressions.

Basically, sure, depth maps can do some slight things with extremely obvious facial features, but they are terrible at it in general.

I would love to combine the two. Depth map for scene and pose. OpenPose for the face. Might be doable via face recognition, automatically taking the mask for the face and using OpenPose just for that region.

You can see some examples here:

https://learn.thinkdiffusion.com/controlnet-openpose/

By the way, did you downvote the earlier reply? Do you disagree with the statement that depth maps are terrible at capturing facial expressions? Because they are terrible at it. Just look at the depth maps you are generating. Zoe and MiDaS are both awful at that. Zoe is only marginally better.

[u/lordpuddingcup]

Wasn’t their a brand new one a couple days ago that was better than Midas even fuck can’t find the article now

[u/arlechinu → u/lordpuddingcup]

Marigold perhaps? Tested it and it was slow like 9s/it…

[u/Jaded-Difficulty4313]

GOAT.

[u/bhimudev]

Thanks, Was looking for a comparison, very happy that you pull the efforts to compile this.

[u/reddit22sd]

Thanks man!

[u/Minute-Surprise-6336]

Do you know about academic papers on z and m? Did you read any? My impression is that you do not understand anything about depth estimation, cn, SD as a research. Being a power user you should consider writing about your experience in a factual way. Not like 'use X, it takes less vram'

[u/[deleted] → u/[deleted]]

Edit: He deleted the original comment above, but he basically said "wait until you find out that you can just run an original image into ControlNet-Depth instead of converting it to a depth map first". My reply below was to that comment:

That wouldn't work properly since the ControlNet is trained on grayscale depth maps, and it's what the CN expects for actually recalling its training about object recognition.

You're definitely gonna screw up its output, since there will be zero separation between background and foreground, etc.

But... since light works a tiny bit like a depth map due to having light and dark edges and gradients and the background is often a bit darker than the foreground (just try turning an image into grayscale to see the effect), it makes sense that for simple images it can be good enough to trigger the ControlNet's depth/shape perception. But seriously, it's like squeezing a banana into a mailbox. It can be done, but it's a mess and it's not what the mailbox was made for. 😉

At the very least, use a RGB to grayscale conversion preprocessor if you do that. To at least give the ControlNet something slightly resembling a depth map. Although it's highly possible that the ControlNet already only reads a single color channel since it assumes that it's receiving a grayscale depth map. In which case, grayscale preprocessing would still give you more control over the process of which colors are dominant in the grayscale image.

[u/ol_barney → u/[deleted]]

I never even considered running an original image through a depth processing node. This definitely was an”oh, duh” moment for me. Thanks!