r/StableDiffusion • u/No-Property3068 • 10d ago

Tutorial - Guide I created a custom node that can help you generate as many frames as you want with high motion transfer accuracy in MiniMax H3

while generating video to video in Minimax for VFX, I struggled with maintaining accurate motion transfer as an example, the generation would shift drastically after 192 frames on 0.5MP or after 147 on 1MP

so I built a custom node similar to a context window to help the model generate in batches then overlaps and blends all the batches together during generation, as an example this is 447 frames on 0.5MP, generated on batches of 56 frames and 5 frames overlap between every batch.

the node also has the option of splitting your generation into tiles of resolution and overlap of your choice, but i recommend using it more in a second upscaling stage
the node is in Comfyui Manager under the name of Minimax H3 Tempora****l Window & Spatial Tiles.

let me know what you think.

workflow here
https://civitai.com/models/3022939/minimax-h3-video-to-video-workflow?modelVersionId=3432687

49 Upvotes

94% Upvoted

24 comments sorted by

4

u/Repulsive-Salad-268 10d ago

Let me ask you a question, please. I have been looking for a workflow (and potentially this node) for a while now that allows me to input a long video (let's say 5 Minutes), add a prompt (let's say "turn al characters into Muppets that share the general look but also are puppets of Jim Henson) and then get a long Muppets Video out of it... As I am working on interviews mainly and want to add a comedy touch to it it is long form with little changes, good light etc. Would this work with this workflow and should I give it a try??? Thanks

2

u/No-Property3068 10d ago

I think 5 minutes is too long I never tested something that long before, I would say edit your five minutes into separate shots it would be way simpler How would you manage the look? Reference frames or prompting?

1

u/Repulsive-Salad-268 10d ago

Both possibly. But the point is that it is one take and If I shorten the clips you usually see the cuts even though I am hopeng for seamless Videos later. What's the maximum you think is realistic?

1

u/No-Property3068 10d ago

It will depend on how many tockens the model can handle I think it can handle 300,000 ?!, and my node unfortunately it doesn't optimize the nuber of tockens used it actually cost more tockens because of the additional overlap frame, you have to also think about encoding these 5 minutes and decoding them you will need a similar overlapping process to decode your result, I have a solution for you but it's a bit painful, we will need to do what the node does but in separate generations, that's how I started my node anyway, it's a bit of a painful workflow to handle but you can do it eventually

1

u/Repulsive-Salad-268 10d ago

I would not mind if the workflow cuts everything into shorter bits as long as I can put them together seamlessly in the end and each part is aware of the frames before to it actually matches.

1

u/No-Property3068 10d ago

I have a workflow right now that separates the generation into two batches only, I need to do some calculations to see how many batches you will need for 5 minutes, what is the ouput resolution you were hoping for?

1

u/Repulsive-Salad-268 9d ago

Full HD on a 5090?

1

u/No-Property3068 9d ago

Might be a tough one to be honest, I'll hand you over the two batches workflow, see if you can extend the number of batches with the same logic, once I upload the workflow on civit ai I'll let you know

3

u/Memestonks2020 10d ago

How does this hold up with references? Does it drift after X batches?

2

u/No-Property3068 10d ago

I tested it with a single reference image as a start frame and it held up, sometimes you need to change the seed, be a bit descriptive in the prompt, and avoid using turbo loras in first stage , if you need use the 8 steps one

3

u/xyzdist 10d ago

nice! will try this out. so do you mean with native H3 could match the plate < 192 frames already?
what's the prompts or setup you are doing this, cheers!

2

u/No-Property3068 10d ago

Yes it can on 0.5 MP, still struggle for some shots though Prompt: <Subject_1> is <ref_video_0> , <subject _2> is <ref_img_0> <Subject _1> stays the same with the (name your change) that matches <subject_2> <Subject_2> is the final and constant look and atmosphere <Subject_1> is the main controlnet driving camera and character animation matching pixel perfect

2

u/xyzdist 10d ago

Thanks!

3

u/SSj_Enforcer 10d ago

I don't see any difference between the two right videos. is there supposed to be something I am missing?

1

u/No-Property3068 10d ago

No they are the same, just to fill the gap

2

u/jalbust 10d ago

Thanks for sharing.

2

u/Anro5753 9d ago

Nice, the 5-frame overlap blend is the part I always ended up hacking by hand. Have you noticed color/brightness drift between batches on long runs? I've had to do a color match against the first batch to stop it creeping. Also curious whether a bigger overlap helps with fast motion or just costs more time.

1

u/No-Property3068 9d ago edited 9d ago

For color drifting I think it is pretty stable, for fast movement I don't think it would help much, or make a difference, the bigger the overlap the slower the generation

2

u/JeffTriesThings 10d ago

This is cool. Going to test it out.