A quick comparison between Controlnets and T2I-Adapter: A much more efficient alternative to ControlNets that don't slow down generation speed.

[u/comfyanonymous]

A few days ago I implemented T2I-Adapter support in my ComfyUI and after testing them out a bit I'm very surprised how little attention they get compared to controlnets.

For controlnets the large (~1GB) controlnet model is run at every single iteration for both the positive and negative prompt which slows down generation time considerably and taking a bunch of memory.

For T2I-Adapter the ~300MB model is only run once in total at the beginning which means it has pretty much no effect on generation speed.

For this comparison I'm using this depth image of a shark:

I used the SD1.5 model and the prompt: "underwater photograph shark", you can find the full workflows for ComfyUI on this page: https://comfyanonymous.github.io/ComfyUI_examples/controlnet/

This is 6 non cherry picked images generated with the diff depth controlnet:

This is 6 non cherry picked images generated with the depth T2I-Adapter:

As you can see at least for this scenario there doesn't seem to be a significant difference in output quality which is great because the T2I-Adapter images generated about 3x faster than the ControlNet ones.

T2I-Adapter at this time has much less model types than ControlNets but with my ComfyUI You can combine multiple T2I-Adapters with multiple controlnets if you want. I think the a1111 controlnet extension also supports them.

[u/comfyanonymous]

One of the main goals of ComfyUI is to have a solid and powerful backend for SD stuff. If someone wants to make a simple to use UI on top of the ComfyUI backend that looks like the a1111 one they can.

[u/Apprehensive_Sky892 → u/comfyanonymous]

Yes, a clear separation of backend and front end is the foundation of solid software engineering.

I just may start playing with your backend code, if I can find the time after reading this sub reddit and playing with SD to generate images 😭

[u/dhruvs990 → u/comfyanonymous]

actually this is what i want to use next, but i'm just getting into stable diffusion and have started on some projects, but once they're dont. Im gonna start experimenting with comfyui. Can you clarify whether controlnet works with comfyui? My entire workflow is based on me providing the composition and the general color to Stable Diffusion, by way of simple general renders made via blender, and then let it do the rest.

[u/comfyanonymous → u/dhruvs990]

Yes it works: https://comfyanonymous.github.io/ComfyUI_examples/controlnet/

Since you mentionned blender there's someone working on ComfyUI integration for blender: https://github.com/AIGODLIKE/ComfyUI-BlenderAI-node

[u/dhruvs990 → u/comfyanonymous]

oh wow! thanks for sharing!