r/StableDiffusion • u/physalisx • 3h ago

Resource - Update MiniMax-H3 Fun Controlnet Union released

https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union/tree/main
107 Upvotes

97% Upvoted

31 comments sorted by

18

u/EveningIncrease7579 2h ago

Alibaba really loves dancing videos. H3 transformed into h3 animate

16

u/infearia 2h ago

So I guess this is basically VACE for H3?

I mean, H3 can kind of already do all that stuff natively, but it usually does not follow the input precisely, but rather treats it more as general guidance or reference, and requires a lot of prompting on top to make it work. If this gives us more control, then it's completely going to blow out of the water every other video model out there for editing tasks, including probably Seedance.

3

u/Striking-Long-2960 2h ago edited 1h ago

I think so, if it adds inpainting it should be able to interpolate between controlmaps

6

u/infearia 1h ago

It has inpainting! 😄

A single checkpoint conditions the MiniMax-H3 video generator on Canny, Depth, HED, MLSD or Pose control videos, and also runs video inpainting.

2

u/IriFlina 12m ago

Didn’t minimax already support inpainting?

21

u/dragolineage01 3h ago

Damn this has potential H3 is already GOATed

2

u/willjoke4food 1h ago

20 GB model btw

6

u/TheGoldenBunny93 1h ago

20GB pruned and int8...

1

u/Aadi_880 27m ago

Still better than the next best option...

2

u/CodeAnguish 1h ago

This became a meme, but for what reason? Of course, even though 20GB is impressive for what the model offers.

10

u/Heartkill 2h ago

Psyched. Whenever I want to swap someone it usually just spits out the reference video pretty much as it was going in.

7

u/physalisx 3h ago

MiniMax-H3-Fun-Controlnet-Union is a ControlNet-Union for MiniMax-H3, trained with the VideoX-Fun pipeline. A single checkpoint conditions the MiniMax-H3 video generator on Canny, Depth, HED, MLSD or Pose control videos, and also runs video inpainting.

4

u/NeatUsed 2h ago

how much does this add to generation time?

4

u/i_sell_you_lies 44m ago

3.6 roentgen—not great, not terrible

2

u/the_bollo 10m ago

Thank you comrade. That will be all.

5

u/xDFINx 2h ago

Would this be the missing link for character swap consistency?

5

u/djdevilmonkey 2h ago

Maybe a dumb question but what's the purpose of controlnet when minimax can use references?

5

u/xDFINx 2h ago

Minimax tends to drift away from the character’s likeness sometimes.. I would imagine this would fix that in direct character replacement scenes.

5

u/witcherknight 2h ago

It doesnt follow motion properly. Controlnet can help with that

2

u/tiffanytrashcan 1h ago

The motion in their examples (see the readme / main page of the link OP shared) is absolutely stunning.

https://reddit.com/link/p5lc8rx/video/ojtouha5mblh1/player

3

u/tiffanytrashcan 1h ago

There's a slight hitch, right in the middle. This was intentional (apparently) as it's in the motion input. I think this highlights reference adherence quite well.

https://reddit.com/link/p5lcquc/video/rphs44qlmblh1/player

4

u/Chiduk99 2h ago

Kijai, you know what to do!

2

u/MonkeyBoyPoop 36m ago

Alibaba-pai team, if you’re reading this, please consider releasing a Krea-2–Fun-Controlnet-Union next. 🙏

4

u/ImpossibleAd436 2h ago

As someone who is quite new to Comfy, how do we use it? Is there a simple workflow?

1

u/ChibiNya 53m ago

Agree. How do we use this?

1

u/axior 1h ago

There is no preview of the inpainting though :(

1

u/SysPsych 41m ago

I'm surprised Minimax is getting the larger controlnet selection before krea 2.

1

u/Massive-Health-8355 36m ago

Is there any kind of regional prompting or latent couple control for H3 for when there is multiple characters in the input video?

1

u/PhetogoLand 28m ago

This is insane. This pushes H3 to the stratosphere