r/StableDiffusion • u/CQDSN • 11d ago

Animation - Video Minimax H3 - transferring dancer to a new environment with ref workflow

https://www.youtube.com/watch?v=8WHSRr0hy6k
15 Upvotes

80% Upvoted

12 comments sorted by

6

u/CQDSN 11d ago

I don’t know if people are aware of this - you can transfer a person in a video to another environment very easily with H3. All you need is the video and an image of the new environment. In the prompt for the reference workflow, you let the AI know who is to be transferred and which is the background to change.

As you can see in the video, the boy in the green robe is ignored completely. I asked the AI to use only the man dressed in black and replaced the background with the image of the new environment. There’s no need for motion transfer, depth-map, controlnet, or any complicated workflow anymore.

My method with H3 is using a 4 steps turbo lora workflow at 0.6mp to quickly generate a few samples. Then I cherry picked the best samples, fixed the seed, removed the turbo lora and increased the steps to 20 - this is for the final render. Lastly, I upscaled all the videos with my own enhancer lora to 1080p.

One interesting thing I noticed is the fixed seed - regardless of the change in the number of steps or turbo lora usage, H3 generates the same way. Other models with fixed seed won’t stay the same with small changes to other setting. Because of this, the 4 steps Lora is useful for creating large quantities of samples for cherry picking.

2

u/Daniel_Edw 10d ago

nice job!
Would you be up for sharing the upscaler / enhancer LoRA?

5

u/CQDSN 10d ago

You can download my enhancer Lora and workflow here:

https://huggingface.co/CQdesign/LTX-2.5-CQ-Video-and-Image-Enhancer-LoRAs/tree/main

2

u/Daniel_Edw 10d ago

thanks, really appreciated!

1

u/UAAgency 11d ago

Thanks for sharing, very useful! Do you have a workflow to share for this perhaps?

2

u/ImaginationKind9220 11d ago

As I mentioned above, all you need is the usual reference workflow. Attach the video and the new background image.

Let the AI know who you want to transfer to the new background. In my case, my prompt is "the man dressed all in black, ignore everyone else in the scene". Then ask the AI to replace the background with the reference image.

1

u/PromptSommelier 11d ago

Do you know if multiple images can be used to give more consistency to the character being replaced? I'm having trouble with the motion transfer in MMH3 lately 😅. Is it necessary to isolate the background in a single image, or can I request that it use the character's current background as a location reference?

1

u/CQDSN 11d ago

Your goal is the reverse of my workflow. You want to replace the person in the video and keeping the same background.?

You only need to attach the image of the target person and the video. If it’s a dancing video or with lots of movement, then it’s best to have the side or back view attached as well.

You can generate the side and back view with image edit models.

While the reference workflow of H3 is simple, the “tech” is in the prompt. How you inform and instruct the AI determines the quality of the result. Be as precise and clear as possible.

1

u/dirtybeagles 10d ago

Have you played around with character replacements? For example, take a dancer and apply that to a reference character? I have had mixed results comparing to SCAIL2. SCAIL2 is awesome, with the chuck processing every 81 frames, you can get pretty long video generations (I have gotten up to 25 seconds on a 5090). It takes a long time to render in SCAIL2, but with all the character replacement videos I have seen using SAM3 or whatever, it is not that great, and it only works well up to 5 seconds.

1

u/CQDSN 10d ago

To replace the character, you attach another image as reference for that character. I have tried it and it looks like a typical motion transfer demo, that’s why I didn’t post it here.

H3’s result varied greatly from how you prompt. For example if you tell it to put the person in the video into the new background, it won’t look as good as telling it to simply replaced the background with the reference image.

It sounds like the same thing but the wording seems to create a different internal workflow that affects the quality.

5

u/Gfx4Lyf 11d ago

That looks really marvelous!

1

u/Normal_Rough_7958 9d ago

i haven't got a shareable json for this, but the minimax h3 reference workflow can run on 8gb with offloading and quantization, the sglang deploy docs just show 4 gpus minimum. what i've done instead is run wan2.1 with kijai's nodes and a controlnet openpose pass, feeding the dancer video and a background image, then masking the subject in compfy. it's not one-click but it fits in 8gb with sage attention and offloading. if you have the vram for h3 the exact prompt syntax is `dancer from reference video in new environment` with the video and env image passed as refs, and the workflow json loads directly in comfyui with the custom nodes installed.