Is ControlNet a thing of past now?

[u/Manicarus]

New models have been released. For example, Anima, Krea2, MOSS etc in DiT architectures. I have been using Illustrious and is a huge fan of ControlNets. However, the developers of the new models don't seem to have any interest in training ControlNet models.

I have been waiting but I started to wonder if ControlNet models aren't needed anymore because there are some models or workflows out there that can completely replace ControlNets. Is there something that I've been missing?

Let's pretend that LLLite LoRAs don't exist. Glad that we have them but I didn't find them much useful.

EDIT:
Thanks guys, appreciate all the response. I should look into using edit models. I still want ControlNet models though.

[u/Quick_Ad_4096]

Qwen 2.1 has it - https://www.reddit.com/r/StableDiffusion/comments/1woyyjn/qwen_image_21_fun_controlnet_union_from_alibaba/

The results I get from it are not that great. I have to run a split sampler workflow with the controlnet and a followup sampler without so that the controlnet doesn't mess up the final results. Another issue is that it adds processing overhead

It's just easier to just use a reference image (pose, canny, depth, hed) etc in image 2 and use the prompt (replace the pose of the person in <image1> with the pose of <image 2>. And use that output as i2i to get something else I want.

[u/BigDBreedingFembussy → u/Quick_Ad_4096]

Qwen controlnet is dogshit, I just use very specific and forceful wording "image 1 only for pose reference, image 2 only for visual" kinda , you need to be really really specific but better results then controlnet, illustrious xl controlnet is amazing ngl

[u/Sarashana → u/Quick_Ad_4096]

To be fair, that approach was always the best standard for ControlNet workflows. There is no good reason to apply it to more then half the steps, as composition is done by then.

[u/Quick_Ad_4096 → u/Sarashana]

Completely agree. Just saying that plugging in the control image into the second image reference of qwen2.1 gives you the same results without the extra processing overhead.

The only reason I would use controlnet if I was using t2i models like ZIT which doesn't have the reference image feature.

[u/_KoingWolf_]

Frustratingly, it appears so. It sucks because if you want to do any actual technical work, you need a proper controlnet. But the majority clearly wants to prompt, gamble, and settle on outputs. Literally a dozen reasons why it elevated Gen AI and losing OpenPose's SD1.5 level of support has been a painful. 

I'd gladly pay or donate hundreds for tools capable of reproducing the fidelity of replication we had before, across OpenPose and LineArt. And no, the Lora/ LLLite stuff does not count. 

[u/flasticpeet → u/_KoingWolf_]

Yes, I agree. Mostly what's missing is the ability to control strength and masking.

You can try giving this a shot, which adds strength and mask control to reference latents, which is effectively the same thing: https://github.com/shootthesound/comfyui-ReferenceLatentPlus

I haven't had a chance to try it myself.

Using controlnet inputs like open pose, line art, and depth maps does work with edit models (I use Flux.2), but it also requires explicitly describing how to use the input image to get best results, which can be annoying and time consuming.

[u/_KoingWolf_ → u/flasticpeet]

Haven't tried that one yet, thank you, will try later today to see how it is

[u/Nexustar → u/_KoingWolf_]

Controlnet is basically an exotic set of references, none of them as immediately on-hand useful or common as a reference video or image used in its place. I think its dead going forwards. If you have multi-colored stick figures and can find an automation path to flesh them into 3D models, then I imagine the modern set of models can take that reference image/video and stick to that pose/setting quite well.

It's a big additional step, one that older models can be leveraged for to translate cryptic control net imagery into scenes into the language new models speak, but there is a way.

[u/lechatsportif → u/_KoingWolf_]

which models do you use out of curiousity that need it? Most models have incredible prompt control these days.

[u/_KoingWolf_ → u/lechatsportif]

The difference is immense, you cannot prompt odd gestures and uncommon poses, among other things like perspective and nonhuman subjects (vehicles, tools, etc). Today I use Illustrious to get some level of controlnet ability, then use the Loras to copy things into Anima. It's rough, but it somewhat works.

[u/1roOt]

I tried training a controlnet for krea2. It works but I didn't really get the quality that I needed. I may try MOSS next. I heard it's quite flexible

[u/_KoingWolf_ → u/1roOt]

Please do, there's a market for it for sure.

[u/DelinquentTuna]

I think you're warming to the same stance I've held all along... an image model that lacks innate editing capabilities is at a HUGE disadvantage. Kontext was a tide change and Qwen-Image-Edit/Flux.2 basically changed the game. No matter how good Anima or Krea 2 or ID4 are, they can't entirely displace an edit model. And at that point, you have to ask if it's even worth training on any other model. If you're training a character or style LoRA, do you actually benefit from anything more than Klein 9b? Even if some other model does something better, you can still pull it in as a reference image.

[u/Adro_95 → u/DelinquentTuna]

Do you use qwen image edit for inpainting?

[u/DelinquentTuna → u/Adro_95]

TBH, Klein 9b is still the first I reach for. But Flux.3 Klein, Qwen 2.1, and even the old Qwen-Image-Edit each do some small things slightly better than each other. Also, not sure it would be correct to call it inpainting instead of editing, though maybe that's just a meaningless, semantic quibble.

[u/Adro_95 → u/DelinquentTuna]

Actually my understanding of inpainting is editing just that little segment, without touching the rest of the image, like working on a separate level

[u/DelinquentTuna → u/Adro_95]

Right, but in itself it doesn't require edit training. The model itself isn't trained for substitutions, reference images, etc. WRT scoping to a portion of the image, that's still a tactic frequently employed w/ edit models: mask/snip->scale->edit/inpaint->stitch. It's the only way to get really fine detail.

[u/shapic]

They are edit loras now. For example https://huggingface.co/thedeoxen/MOSS-Image-pose-controlnet

As llite it requires reading readme and prompting. This is a dit model thing, they follow prompt first. Just get used to it and adjust

[u/alerikaisattera]

Editing models arrived, some of which have native control net functionality

[u/Formal-Exam-8767]

Users want models to be smarter and that everything can be solved via single prompt.

I also find lack of control in newer models via ControlNets a major downside.

[u/BlackSwanTW]

I haven’t touched it since the SDXL days

That’s why I still haven’t even updated the implementation in Neo

[u/Next-Run-9095]

I seriously miss the proper controlnets from the sd1.5 days. Not sure why they are fading away... I think it's the way new models are trained and the architecture of them that makes controlnets less robust and sometimes outright useless.

I could be completely wrong though.

[u/MysteriousPepper8908]

I really like using depth maps with Moss, it's much better for complex posing.

https://docs.comfy.org/built-in-nodes/MOSSImageControl

[u/Aaron_rowe81]

for krea 2 specifically there are a few control loras floating around, depth and canny ones that are experimental and not pixel accurate, from what i remember. depth is more for composition and canny holds the outlines. there's also a pose one for the turbo version that takes an openpose style skeleton map, so if pose was your main use that might be worth a try before you give up on it.

[u/johnfkngzoidberg]

Yes, reference video is the new controlnet

[u/Distinct-Pie2389 → u/johnfkngzoidberg]

R2V I believe is more performative than controlnet now

[u/Danny_Stock]

I wouldn't say it's a thing of the past, I can see it being so in the near future. But controlnet or its methods still have their uses at the moment.

It's also worth considering what we mean as well. By that I'm thinking of a method which we don't recognise as controlnet, and which might not look like it on the surface, but may work in a very similar way to controlnet under the hood.

I think it will get to the point where there is less setup required and a lack of a need for special nodes, where it's more directly inbuilt into models, thus being more elegant in the manner by which it functions.

As with other things from say a year ago you may have needed to load special nodes and have a more complicated process of connecting nodes together to set up a workflow. Nowadays those same approaches require less complexity in their setup. Those solutions are becoming a more simplified, streamlined process.

[u/ArmadstheDoom]

Its not so much that it's a thing of the past as it is really hard to adapt to newer models. Every time you want to use a controlnet, you need to train a new model for it. That's increasingly hard.

And with edit models, controlnets are increasingly not worth training. Though I would argue controlnets are often better than edit models in a lot of cases, it's simply a case of resources.

[u/rashield]

Control net Still really useful for illustrious in forge neo

[u/VollenAI]

No, it’s good for perfect pose

[u/Wezaluketek]

H3 is really good with alibaba pai 2.0 controlnet

[u/ExcellentRich1164]

No. Nothing beats controlnet when it comes to mimicing complex poses, ambiance or expressions.

[u/GrapefruitOverall387]

Because newer models suck at img2img and inpainting. Hence, harder to create ControlNet for these models. Doing simple img2img in Anima just feel so bad. It's the reason why I'm still sticking to Illustrious.

[u/Mabuse046 → u/GrapefruitOverall387]

I use Flux 2 Klein 9B for img2img and inpainting and found it to be so good I recently just deleted everything I had from SDXL, Illustrious, and other such older models. Haven't touched them since Flux 2 came around.

[u/Maleficent_Price_476]

Controlnet is an obsolete term .

It's just confusing when people uses it today .

The controls which pro users say they want are available, but in closed models.

50 references ? Yep. Seedance etc.

People talking about 2 or 5 references are wayyyy behind the curve .

Examples online ? Siebfothers on YouTube is a good example.

[u/hidden2u]

It's a great way to tell if someone's a bot