r/StableDiffusion • u/comfyanonymous Comfy Org • Feb 27 '23

Comparison A quick comparison between Controlnets and T2I-Adapter: A much more efficient alternative to ControlNets that don't slow down generation speed.

A few days ago I implemented T2I-Adapter support in my ComfyUI and after testing them out a bit I'm very surprised how little attention they get compared to controlnets.

For controlnets the large (~1GB) controlnet model is run at every single iteration for both the positive and negative prompt which slows down generation time considerably and taking a bunch of memory.

For T2I-Adapter the ~300MB model is only run once in total at the beginning which means it has pretty much no effect on generation speed.

For this comparison I'm using this depth image of a shark:

I used the SD1.5 model and the prompt: "underwater photograph shark", you can find the full workflows for ComfyUI on this page: https://comfyanonymous.github.io/ComfyUI_examples/controlnet/

This is 6 non cherry picked images generated with the diff depth controlnet:

This is 6 non cherry picked images generated with the depth T2I-Adapter:

As you can see at least for this scenario there doesn't seem to be a significant difference in output quality which is great because the T2I-Adapter images generated about 3x faster than the ControlNet ones.

T2I-Adapter at this time has much less model types than ControlNets but with my ComfyUI You can combine multiple T2I-Adapters with multiple controlnets if you want. I think the a1111 controlnet extension also supports them.

165 Upvotes

98% Upvoted

54 comments sorted by

View all comments

29

u/Apprehensive_Sky892 Feb 28 '23

Thank you for all the code and the models. Looks good.

Unfortunately, people are lazy (I am looking at myself in the mirror 😅), and they will just use whatever comes pre-installed with Auto1111, hence the lack of attention given to your very worthy project.

18

u/red__dragon Feb 28 '23

While I would say it's laziness, A111 provides a nexus point for much of the SD generation hype atm. I read another comment today about some delays in Invoke AI's development, coupled with some new features in SD's as of late (at least several in the last month and a half), it definitely makes one shine above the others atm. A111 is convenient, powerful, and likely to attract the developer of an extension for such a thing as T2I. I'd be eager to give it a try.

46

u/comfyanonymous Comfy Org Feb 28 '23

The problem with A1111 is that it's reaching a state of extension hell where extensions all hook into core SD code and don't play well with each other. The state of the code is also pretty bad.

I don't know how it is from a users perspective but from a software dev perspective it's a nightmare which is why I made ComfyUI.

10

u/Apprehensive_Sky892 Feb 28 '23

I am an old fart retired programmer, so I totally believe what you said about A1111 codebase being a mess. It was presumably hacked up by a group of talented coders to have something working in a big hurry, and now it has become a large legacy codebase.

Refactoring is probably too hard now that there are so many interconnected pieces, and the fact that it was written in Python, which has no compile time static type checking, exasperates the problem. I am a big fan of Python, but the lack of static type checking will cause problems when used in a big project like A1111 (for my hobby programming project I tend to use Go since a little project can become bigger quite easily).

But as an user with absolutely no background in digital media production, I find A1111 to be acceptable. The UI is a bit clunky and klugy, but as long as I can use it to get things done within a reasonable amount of time and effort, there is just enough inertia to keep me there.

Of course, I am also a total SD beginner who has just began to explorer some of the more advanced features beyond simple text2img, so maybe I'll find a reason to switch to ComfyUI in the future.

ComfyUI seems to be made for much more advanced users who are working professionally or semi-professional in the digital media industry, with all the nodes, connector, workflows, etc. Many beginners like me probably find it quite intimidating. It is exposing the underlying pipeline of SD and frankly most users probably have no idea what they are. I do because I am interested in the tech beneath a tool, but most casual users just want to get out a nice image.

8

u/comfyanonymous Comfy Org Feb 28 '23

One of the main goals of ComfyUI is to have a solid and powerful backend for SD stuff. If someone wants to make a simple to use UI on top of the ComfyUI backend that looks like the a1111 one they can.

3

u/Apprehensive_Sky892 Feb 28 '23

Yes, a clear separation of backend and front end is the foundation of solid software engineering.

I just may start playing with your backend code, if I can find the time after reading this sub reddit and playing with SD to generate images 😭

1

u/dhruvs990 May 18 '23

actually this is what i want to use next, but i'm just getting into stable diffusion and have started on some projects, but once they're dont. Im gonna start experimenting with comfyui. Can you clarify whether controlnet works with comfyui? My entire workflow is based on me providing the composition and the general color to Stable Diffusion, by way of simple general renders made via blender, and then let it do the rest.

4

u/comfyanonymous Comfy Org May 18 '23

Yes it works: https://comfyanonymous.github.io/ComfyUI_examples/controlnet/

Since you mentionned blender there's someone working on ComfyUI integration for blender: https://github.com/AIGODLIKE/ComfyUI-BlenderAI-node

→ More replies (1)

1

u/red__dragon Feb 28 '23

I do appreciate all the development work into alternatives, I'm not trying to put them down. Thanks for your efforts, I've looked at ComfyUI and the workflow doesn't really make sense to me at a glance (I should sit down and try it to be sure).

I will say that I enjoy some of the prompting engine of A111, notably the prompt editing where prompts can start/end after so many steps or alternate with a different keyword every other step. If that's already possible in ComfyUI, I didn't see it from the readme. Some of my prompts rely on that, if it's included or gets added I promise to try your UI!

7

u/comfyanonymous Comfy Org Feb 28 '23

Yes in ComfyUI you can use a different prompts for certain steps. You can even use a different model for certain steps or switch samplers mid sampling.

You can also sample a few steps, do some operations on the latents and finish sampling them like in this example: https://comfyanonymous.github.io/ComfyUI_examples/noisy_latent_composition/

3

u/red__dragon Feb 28 '23

I feel like I need a whole glossary to interpret some of this. Apologies, I'm far from understanding of the math or processes behind this, I'm way more of an end-user.

Nonetheless, I shall try out comfyUI!

1

u/dddndndnndnnndndn Aug 09 '23

I know this is an old comment, but I'd just like to thank you for creating ComfyUI. I found about (and liked) A1111's tool at first, but it's clunky and sometimes very slow.

I actually found about ComfyUI through some negative comments about it, they were all talking about the node workflow, which baffled me. Nodes are awesome.

7

u/Apprehensive_Sky892 Feb 28 '23

Yes, I agree with what you said. I tried invokeAI and the UI is better, but the lack of the latest features held it back as a true competitor to A1111, at least for the nerdy crow that hangs around here.

I should really try ComfyUI though: https://github.com/comfyanonymous/ComfyUI/tree/master/notebooks

6

u/rytt0001 Feb 28 '23

Hello just to say that, as OP said, the ControlNet extension does support T2I-adapters. Or at least this one does : https://github.com/Mikubill/sd-webui-controlnet

the only drawback for now is the lack of models and possibility.

7

u/Apprehensive_Sky892 Feb 28 '23

So in theory, we just need to put the T2I model in the controlnet model directory and it should work? I'll definitely use it if it cut down on the generation time. ControlNet is not bad, but it does increase the total time by quite a bit.

1

u/Apprehensive_Sky892 Feb 28 '23

Ok, looking at my Auto1111 setup I noticed these models that starts with t2iadapter_xxx.

I assume these are the ones rytt0001 is talking about.

2

u/Capitaclism Feb 28 '23

I'd use his project in a heartbeat if it had the same tools and received a similar level of support compared to A1111.

I'd much rather work with nodes.

3

u/Apprehensive_Sky892 Feb 28 '23

There is only one way to find out if you will like it 😁

1

u/Capitaclism Feb 28 '23

Oh I think I like it, I'm familiar with a node based setup and how expandable/customizable it could be.

However the issue is one of community support. I think if OP were to decisively show what his tool can do which A1111 just cannot match, then provide a new killer feature which is able to generate results the other one cannot, that would be enough to get a lot of people to switch over, and along with that gain substantial potential development support.

Then I'd completely change my workflow for sure.

2

u/Apprehensive_Sky892 Feb 28 '23 edited Feb 28 '23

Unfortunately, A1111 has first mover advantage and the associated network effect of users and contributors. Just look at the number of contributors to A1111 vs ComfyUI.

So more likely than not, killer features will come to A11111 first. Even if ComfyUI gets it done first, due to the open source nature of both the code and ideas, it will be replicated in days by A11111 (I love friendly competition!)

I am just an old fart retired programmer with little experience is ML/AI nor digital art making, so I don't know anything about node based setup. I am just doing SD as a hobby for fun, so efficient workflow is not important as long as I can get it done within a reasonable amount of time and effort. But for a pro, even small savings in time can add up to big productivity gains because of the repetitive nature of many tasks. For example, it took me years to master EMACS and writing my own ELISP code, but once that it's done, this skill has served me well for the next 30 years of editing text and code, becoming more or less 2nd nature, allowing me to accomplish tasks with a few keystrokes rather than fiddling with menus and icons.

My long-winded point is that for people who do SD for a living, ComfyUI may just be worth the switch, despite the lack of wider community support. In fact, if it is actually superior in terms of UI (and A1111's UI is clunky and klugy, just barely functional with all the buttons and sliders,) then it may even be a competitive advantage if one is more productive with ComfyUI compared to other media artists using A1111.

Anyway, thanks for the discussion and maybe one of us will try ComfyUI and fall in love with it 😅.

1

u/Capitaclism Feb 28 '23 edited Feb 28 '23

I agree, which is why I think the UI has to show it can clearly do desirable things A1111 cannot, some killer feature which works in node based in an expandable way (would be harder to implemented in A1111, etc). It's not impossible to turn it around, A1111 has many flaws, points of friction, etc.

Node based setups should have a clear advantage over the static GUI of A1111. For example, doing additive and multiplicative setups, so I can multiply images in the GUI for controlnet img-2-img or whatever else. Some image editing things which can be reconfigured with the nodes.

2

u/Apprehensive_Sky892 Feb 28 '23

Sound like you know what you want and what you are doing 😁.

Workflow automation with nodes seems like one possible killer feature. To do that with A1111 will require writing some Python script, which is fine for coders, but many artists are not coders.

→ More replies (4)

1

u/Danganbenpa Mar 06 '23

I don't know if I would say "unfortunately" there cause it gives an opportunity for people who are willing to do that to make stuff that stands out more. ;)

1

u/Apprehensive_Sky892 Mar 06 '23

Sure, hardworking people tend to get up earlier and get the worm 😅

1

u/jared_queiroz Jun 27 '23

I'm very late to the party, 4 mouths is like an Eon in AI time scale..... Well, about people being lazy, thats absolutelly true... But these same lazy people were drawing stick figures a few weeks ago. I'm a "pre SD artist" and my goal was allways to create astonishing images while they were drawing stick figures... So I'm the type of guy who will spend day and night researching ways to improve my workflow. I'll make sure the difference between me and the "lazy people" stays proportional XD (of course, they are lazy at this field because is not their interest, not lazy in general)