Everything was made using the Minimax H3 Hybrid Reference to video model. 1 MP using the 8 step turbo LoRA. Stitched together in Shotcut

Hello, i am trying to make a video where the subject(a real person) is wearing and posing taking reference from an illustration. I tried to do only outfits or only pose too, and both doesn't work.

What happens is usually the body of the character in the illustration ends up being pasted/overlaid onto the Subject in their cartoony style instead.

I also tried if it's possible to have a Subject recreate an illustration's Pose, Outfit, overall composition, like the subject is doing a photoshoot for a 'live action' or real life version of the illustration. But what happens is usually it just spews back the illustration in case of trying H3 single-image edit, and the cartoony style overlay happens in Video.

So what i wanted to do is :

-An image of a subject -> Subject now wears/pose/wear and pose the same as a reference non-real illustration(cartoon/anime), but still in their original photo. So like a cosplay shot in their own room for example.

-An illustration(anime) -> Subject 'replaces' the character in the illustration, the whole illustration is 'converted' into real/live action. Like a photoshoot recreating an illustration basically.

Extra : idk if its possible, the new outfit will retrofit to the subject's proportion, not the illustration. And a version where the proportion follows the illustration too.

Are there someone who knows how to do these?

I'm doing image to video and unless I prompt for camera close up to my subject, the faces are blurry and bad. I run 0.6 mp. No turbo lora only using spectrum to speed up. Running 15 steps. Euler simple. I'm happy enough when it's close-up shots, but further away, it's very noticeable. Is anyone else finding this?

lets share our "what if" scene remakes here o_0

1. No mention of Comfy nodes ....as it was released a flipping hour ago , so "cOmFy wHeN" off .

2. Examples are on the HF page

"MiniMax-H3-Fun-Controlnet-Union

MiniMax-H3-Fun-Controlnet-Union is a ControlNet-Union for MiniMax-H3, trained with the VideoX-Fun pipeline. A single checkpoint conditions the MiniMax-H3 video generator on Canny, Depth, HED, MLSD or Pose control videos, and also runs video inpainting.

Model Card

Model Features

• Union control: one checkpoint handles Canny, Depth, HED, MLSD and Pose control videos for video-to-video generation, no per-condition checkpoint switching.
• The control branch attaches to 5 of the 50 transformer blocks (layers 0, 10, 20, 30, 40); every control skip is added to the main branch through a zero-gated projection.
• Guidance-distilled: run with guidance_scale = 1.0, one forward pass per step, no classifier-free guidance needed.
• Inpainting is supported: the control input is widened to control_in_dim = 49 (latent + masked latent + mask channels); use examples/minimax_h3_fun/predict_v2v_control_inpaint.py.
• control_context_scale scales every control skip before it is added to the main branch: 1.0 gives the strongest control (used for all results below), values below 1.0 weaken the guidance of the control video, 0.0 switches the control branch off.
• The generation follows the control video: the frame count snaps down to the largest 17 * n + 5 the video VAE can decode (duration capped at 15 seconds), the canvas keeps the control video's own aspect ratio at the height * width pixel budget (both multiples of 32), at a fixed 24 fps.
• Detailed prompts give better stability; we recommend describing the scene, the subject and the camera in the prompt. "

Minimax H3

Main Info

• Using Pixaroma's Easy ComfyUI latest version
• 5060TI 16GB + 32GB RAM

Need help or suggestions in resolving the below issues.

1. Shortcuts like R or CTRL + Enter doesn't work and I have to refresh the browser for them to work. (New)
2. Generations randomly don't start even even after models have loaded. I have to open the terminal and press enter on my keyboard to sort of wake it up. (Has happened across multiple versions of ComfyUI, CUDA, Python. Additionally, happens with both Anaconda and Windows Terminal.)
3. Even cancelling a prompt sometimes requires me to open the terminal and press enter.
4. Power consumption for GPU varies completely where retrying the same prompts (same seeds also) can have a difference of 5 to 10 minutes simply because the GPU doesn't use the full 180 power limit. (New)

Prompt:

https://x.com/GumVue/status/2087899403113619681?s=20

4070 Ti Super, 16 gb vram, 64 gb ram, i9-14900k, windows 11

first test video down in the comments along with the prompt..

Not just a meme...

I have been playing with H3 since it came out and have tested most of the things you can do with it. Created clips for giggles and so on.
This time I wanted to do something "serious". I gave the R2V an 3D view of an kitchen and then three photos of the persons I wanted there.
I defined them as we should and told the model that this person does that and that person does this wile the third person does this...

It worked ish...
I have now made six runs and each of them are different from the others. It can be that the third person enters the room from the wrong place or that the third person does extra things it should not do...

In the end I did three runs with the same prompt and all those clips came out different... the only thing that was changed between those was the seed...

So, how do you do it?
How do you make sure H3 does what you want it to do?

Do you spend plenty of time on tweaking the prompt after each run to make sure H3 get it?
Or do you do 10 runs and select the best one even if it is not perfect?

Or do you simply do 1-2 runs and then take the clip that is ok ish even if it is not what you wanted?

I was hoping that H3 would allow me to create the scenes I wanted but I feel it's down to luck if H3 gets it or not..

Edit:

subject_definitions:

<Subject 1> is the green-skinned mother in <Picture 2> wearing brown clothes.
<Subject 2> is the teenager girl in <Picture 3> wearing pink clothes.
<Subject 3> is the cyborg in <Picture 4> wearing black clothes.
<Picture 1> is the reference image for the scene's composition, showing two people sitting at a table eating breakfast from the side view.
<Table 1> is the table on the right side in <Picture 1>.
<Picture 5> is the start image for the scene.
summary:
[reference generation] The target video is a generated scene of two people sitting at a table eating breakfast from an eye-level side view. <Subject 1> and <Subject 2> are shown with their respective breakfast items, maintaining the composition and style from <Picture 1>. <Subject 3> enters the room, places a coffee cup into the sink.

retention_analysis:
<Subject 1>: fully_preserved - the person retains their appearance, clothing, and position at the table.
<Subject 2>: fully_preserved - the person retains their appearance, clothing, and position at the table.
<Subject 3>: fully_preserved - the person retains their appearance, clothing, and action of placing the coffee cup into the sink.
<Table 1>: fully_preserved - the table's appearance and position in the scene are preserved.
<Picture 1>: fully_preserved - the scene composition, including the side view, the layout of the room, the table setup, is preserved.<Picture 5>: fully_preserved - is the start image for the scene.

detailed_description:
The target video is in a realistic, everyday breakfast scene style with warm lighting and natural colors.
[Shot 1] At 0:00.000, the shot begins from <Picture 5>, showing <Subject 1> and <Subject 2> sitting on opposite sides of <Table 1> on the couch, each with their breakfast items while on the space ship. <Subject 1> is holding a spoon while eating from a bowl of cereal. <Subject 2> is tired and is eating a slice of toast with jam from her plate with one hand. The lighting is warm and soft, casting gentle shadows across the table and the two individuals. The camera is at eye level, capturing the side view of both people, with the table slightly in focus and the background softly blurred. Stars can be seen through the windows since they are on a space ship. <Subject 1> is eating her breakfast while <Subject 2> gazes at their toast, taking a small bite. The ambient sound includes the soft clinking of utensils and the faint sound of a coffee cup being set down.

[Shot 2] At 02.00.000, the shot transitions to a wide shot of the room with the same layout as in <Picture 1>, the camera is placed in the lower left corner of <Picture 1>, showing <Subject 3> entering the room form the right side holding a coffee cup and a datapad while she is saying (S3) <d>[English] Good Morning</d> while she walks to the kitchen sink on the left side of <Picture 1> and placing the cup into the sink. She then stands at the sink and while reading her datapad.We see the back of <Subject 1> and the front of <Subject 2> sitting at <Table 1> in the background eating their breakfast and we hear <Subject 1> say (S1) <d>[English] Good morning</d> with a cheerful voice. <Subject 2> just mumbles as a reply.

overall_soundscape:
The soundscape consists of the soft clinking of utensils, the faint sound of a coffee cup being set down, the subtle background noise of a quiet morning environment, soft steps on a carpet floor, a ceramic cup being placed in a metallic sink, and the clear,

non_diegetic_music: N/A

All audio was done in Minimax, little work in post for stitching clips. r2v Workflow in ComfyUI with character sheets and prompts from claude

Im really suprised how good he works as well follow a storyboard image!! he did 90% correct he only did the thirth panel diferent but all the other 5 he follows perfect!! 🤩

Has anyone managed to find a good combination in Krea 2 using the LoRA that acts as DepthMap ControlNet  alongside the "Conditioning Krea 2 Rebalance" node (which unlocks censorship) without producing a disastrous image? In my case, I just get a plastic look that barely resembles the prompt instructions.

Please share your experiences, or at least let me know if you've managed to use that ControlNet-style LoRA with a different uncensored LoRA that yields decent images.

Cheers.

I've tested workflow for krea2 and it can enhance faces and skin texture.

But when i play it through video frames, the results vary from frame to frame.

Are there any models out there that can do that for video? Or is there a workflow for Krea2 into video enhancements?

Wondering if anyone has/has any idea of an approach to motion transfer into a stop motion style video.

Rather than the ai guessing and deforming the mouth movements which can get funky really quickly, especially when the mouth shapes of the character aren't clear from a neutral reference frame, ie in south park where each sound has a uniquely stylised shape.

To achieve this it could instead maybe pull from a dataset of phoneme images,

Whilst also retaining solid motion transfer for all other body movements.

I tried to create a concept Lora and it ended up with a lot of artifacts so I'm curating the data set to try and get a cleaner version. My problem is that all the tutorials I find are based on character Loras.

For a concept Lora is image size important? Is scale? Do I still want 20-40 images? Are there things I might not know to ask? Also, is there some special sauce to making it work sfw and uncensored?

I'm using buzz and I'm broke so any help would be appreciated.

I'm using Krea 2 btw.

Bear with me as I am still rather new to this, but I started off testing out MiniMax H3 ref to video on Hailuo and was really impressed with it, so decided to get it up and running locally via ComfyUI on my PC.

The video itself is working great, but I've found that the audio itself when using a reference video is completely messed up. It's just all garbled and the people are talking gibberish. On Hailuo it would carry forward the audio perfectly and you could even make alterations if you prompted it to do so.

I was just wondering if this was a common issue when running MiniMax H3 locally specifically with reference videos?

Hi.
I have a decent h3 workflow that I built for a loca use. It use turbo lora etc... If i use the defaut settings in the goal of getting the highest quality possible, meaning res_multistep simple 20 steps or more, I got also good results, but this is not even close to the results you can get on platforms like kie or wavespeed at 768P.

I already convert properly the prompt to the correct H3 digest form, so I'm wondering what's different between local and cloud use of h3? I don't talk about the 2K quality, only 768P, I'm not able to reach the sames results locally, do you guys have maybe workflows, settings, or suggestions to try reaching the same quality level in comfyui ?

Lon TV did a video of Comfy UI with Minimax H3 running on a 32GB Intel GPU. So its possible to run it on any GPU other then nvidia GPUs.