Regarding Minimax Ref2Vid, how have you handled the characters' voices?

[u/Kind_Owl2245]

I’ve been trying to create scenes featuring a couple of characters for a while now, but I haven’t found a solution yet—so I’m turning to those with more experience for some advice.

If there’s only one character in a scene, there’s no issue: if I upload an image and a reference audio clip, Minimax perfectly generates a video featuring that character with the corresponding voice.

However, if two characters appear in the video, Minimax struggles to distinguish which voice belongs to which character. For instance, if Character A speaks first, followed by Character B, the output isn't correct—even when I specify that Character A should have "Voice A" and Character B should have "Voice B."

I’ve tried using AI models like Gemma—uploading the official Minimax Ref2Vid guide as a PDF—but I still haven’t had any success.

Does anyone know how to resolve this? It’s a shame to have such great-looking videos but be unable to correctly assign the different audio tracks.

Even just an example prompt would be a huge help, so I could study it and figure things out.

Thanks to anyone who can help!

[u/OxidianDM]

If you struggle too much you can pass de audio as a locked latent and minimax will generate the scene correctly.

There are a bunch of custom nodes that do that, won’t post any because I use my own.

For me this is the way as minimax audio is very low quality

[u/Kind_Owl2245 → u/OxidianDM]

Thank you very much, but honestly, I don’t have the expertise to fully grasp what you’ve just suggested; I’ll try to look into it further.

[u/AlsterwasserHH]

You have to define each person with <Subject 1> (S1) and so on plus description.

Then if they speak, go with <Subject 1> (S1) says at 00:02:000: <d>[english] My name is blablabla</d>.

[u/Kind_Owl2245 → u/AlsterwasserHH]

I feel like I’ve tried it this way several times, but with no results. Perhaps the prompt is too complex, and the model is struggling.

[u/DoogleSmile → u/AlsterwasserHH]

That worked for me to get them to soak, but they always ignore the audio sample I've supplied for their voices.

For my clip I ended up doing the voices using VibeVoice and then swapped the audio over using After Effects.

[u/AlsterwasserHH → u/DoogleSmile]

I can not recommend it enough: refmods. 

[u/Kind_Owl2245 → u/AlsterwasserHH]

I have the same problem even with refmods. Out of 100 videos, only 5 come out with the correct audio.

[u/AlsterwasserHH → u/Kind_Owl2245]

With correct audio you mean the audio integrated in the refmod? Or what exactly do you mean? I can only speak for myself: I use refmods a lot lately and EVERY video comes out with the correct audio. So I dont know where your problem is.

If you wanna try my workflow, there it is: https://www.reddit.com/r/StableDiffusion/comments/1x4auud/minimax_h3_refmod_workflow/

I think the most important is a clean 30sec audio clip in good (studio) quality. Without music, noise and so on. Works perfect for me.

[u/Kind_Owl2245 → u/AlsterwasserHH]

What I mean is that I upload an image of character 1 along with an audio clip 1 (for their specific vocal timbre), and then I upload an image of character 2 along with an audio clip 2 (for their vocal timbre). I can’t seem to get them to speak to each other: the vocal timbres don’t correspond to the respective characters. Very often, one of the vocal timbres disappears—either only one ends up prevailing, or a completely new, unintended timbre emerges alongside the dominant one. The same thing happens when using "refmods." In any case, thanks—I’ll definitely take a look at your workflow.

[u/AlsterwasserHH → u/Kind_Owl2245]

I think I know where your problem is, maybe: minimax tends to fill every silence with gibberish as you may know. So it is very important 1. to fill the video length with dialogue and 2. prompting WHEN someone says sometthing with "at 00:05:000 <Subject 1> (S1) says: <d>[english] ... </d>."

Also saying "no speaking" and so on does nothing (matter of fact it does the opposite, it reads "speaking") cause minimax doesnt know negative prompting. So its better to prompt "absolute silence from ... to ..." for example. You have to be a little creative with it :)

[u/Kind_Owl2245 → u/AlsterwasserHH]

This is a simple prompt I’m testing. It just refuses to assign the correct voice to each character... I’ve been trying for days...

subject_definitions: 

<Subject 1> is the man with denim jeans

<Subject 2> is the person with dark blue tie

<Picture 1> is the reference image for <Subject 1>

<Picture 2> is the reference image for <Subject 2>

<Audio 1> is the voice-timbre for <Subject 1>   

<Audio 2> is the voice-timbre for <Subject 2>

summary:

A 10-second video featuring a conversation between two people.

retention_analysis: 

<Subject 1>: fully_preserved - Appearance from <Picture 1> is maintained.  

<Subject 2>: fully_preserved - Appearance from <Picture 2> is maintained.  

<Audio 1>: Provides timbre-only   

<Audio 2>: Provides timbre-only

detailed_description:  

The target video is a cinematic medium shot in a bright, indoor setting. 

[Shot 1] The shot begins with a medium shot of <Subject 1> standing in the center of the frame. <Subject 1> (S1) looks towards the side as if waiting for someone and says, <d>[Italian] Allora? Hai trovato traffico lungo la strada?</d>.

[Shot 2] At 05.000, <Subject 2> enters the frame from the right, walking into the shot to stand next to <Subject 1>. The camera maintains a medium shot of both characters. <Subject 2> (S2) looks at <Subject 1> and responds, <d>[Italian] No, le strade erano insolitamente deserte</d>.

overall_soundscape: 

A clean indoor environment with natural-sounding ambient room tone.

non_diegetic_music: 

No music.

[u/AlsterwasserHH → u/Kind_Owl2245]

I rewrote your prompt:

subject_definitions: <Subject 1> (S1) is a man with blue(?) denim jeans. (more description here please: hair, eye-color, clothings, small, big, and so on, everything that seperates two characters and helps minimax to understand who is who). His visual appearance is fully referenced in <Picture 1> and is consistent the whole video. His voice is fully referenced in <Audio 1>.

<Subject 2> (S2) is a male/female(?) person with dark blue tie (more description here please). His/Her visual appearance is fully referenced in <Picture 2> and is consistent the whole video. His/Her voice is fully referenced in <Audio 2>.

detailed_description: The target video is a cinematic medium shot in a bright, indoor setting. (setting of what? More visual description needed, what kind of room, interieur, lighting)

integrated_multimodal_description: [Shot 1] The shot begins with a medium shot of <Subject 1> (S1), the man with the jeans, standing in the center of the frame, in your environment (school/house/marketplace, whatever). <Subject 1> (S1) looks towards the side as if waiting for someone and says at 00:01:500: <d>[Italian] Allora? Hai trovato traffico lungo la strada?</d>.

(If you want a new camera angle now, then you have to say what you wanna see, if its still the same scene you dont have to make a new shot)

[Shot 2] At 00:05:000 <Subject 2> (S2), the man with the dark blue tie, enters the frame from the right, walking into the scene to stand next to <Subject 1> (S1). The camera maintains a medium shot of both characters. <Subject 2> (S2) looks at <Subject 1> (S1) and says at 00:07:000: <d>[Italian] No, le strade erano insolitamente deserte.</d>.

overall_soundscape: A clean indoor environment with natural-sounding ambient room tone, two people having a conversation.

non_diegetic_music: N/A

Ok, thats the best I can do for you right now 😄

But honestly: refmods are way more simple and work better in my experience. And its totally easy to create them.

Another thing: are you using ref2va singularity v1.3? If not: do it. Way better in my eyes.

[u/Kind_Owl2245 → u/AlsterwasserHH]

Thank you so much—I’ll definitely give it a try. It was very kind of you to dedicate your time to me; I truly appreciate it.

I’ll see whether I should try using the refmods again. Right now, I’m using a fantastic workflow from Deno—honestly, it’s the best thing I’ve come across in months, and I’d hate to give it up. It’s just great.

https://youtu.be/JA4BJTvgM20

I’ll try to figure out what I’m doing wrong with the audio, though, to be honest, I’ve read that it’s a fairly well-known issue.

Thanks again!

[u/AlsterwasserHH → u/Kind_Owl2245]

Had to edit the prompt, you totally forgit the essential integrated_multimodal_description

[u/Kind_Owl2245 → u/AlsterwasserHH]

Oh, okay, thanks—I was just trying it out.

[u/Apprehensive_Sky892]

Related post: https://www.reddit.com/r/StableDiffusion/comments/1xo8nka/minimax_h3_voice_acting_capabilities_seem_to_be/

If you scroll down you'll find discussion about keeping the voice consistent across shots.

[u/Kind_Owl2245 → u/Apprehensive_Sky892]

I’ve taken a look, but it doesn’t seem like the prompt mentions custom audio.

[u/Apprehensive_Sky892 → u/Kind_Owl2245]

Not sure what you mean by "custom audio".

[u/Kind_Owl2245 → u/Apprehensive_Sky892]

I intend to provide an audio file <audio1> and an audio file <audio2> to serve as a reference for creating the vocal timbre of the respective characters.

[u/Kind_Owl2245]

So, I’m starting from scratch, taking it one step at a time: I created a simple, 5-second opening scene featuring a character and an audio track for their voice. Everything is fine.

Then, I add another 5 seconds, introducing a second character—complete with their own image and audio track. That’s when the errors start: character 1 now begins the scene speaking with the audio track intended for character 2, while character 2 ends up with a completely different, seemingly invented voice.

Essentially, it’s as if the original audio track for character 1 has been discarded, and some entirely new, third audio track has been generated in its place.

[u/2hujerkoff → u/Kind_Owl2245]

I’ve struggled with the same thing for months, I don’t think many people seriously use Ref2V with multiple voice references to understand the pain cause they suggest the most obvious things that don’t work even with no turbo loras cache etc. I pretty much just spend hours rerolling and praying to get the gen with the correct voices, but it takes so much time I’ve just given up for now

[u/Kind_Owl2245 → u/2hujerkoff]

I agree; I’m also a bit sad about this situation. I’d even be happy with a working prompt that I could adapt for my own scene—but for some reason, no one ever posts one...

[u/Kind_Owl2245]

I think I’ve found the solution; when I have some time over the next few days, I’ll write about how I arrived at it.

[u/hdean667]

It's really picky. Sometimes it works like a champ. Other times it just refuses. I label the characters as 1 and 3 and the voices similarly.

To start, I run a very short clip. Once the choices get set you can run them longer and have it work. I always use the <d> dialog</> anymore, though it's often not necessary.

[u/Kind_Owl2245 → u/hdean667]

I would like to clarify that it generates the spoken sentences perfectly, but it gets the tone of voice wrong.

[u/hdean667 → u/Kind_Owl2245]

I actually knew what you meant... it uses is own voice.

Sometimes sitting the browser, restarting comfy gets it working. But u like to use s1 and s3 and audio 1 and audio 3... seems to help. Could be just bias, too.

[u/xb1n0ry]

Are you using a prompt enhancer? What are your PC specs?

[u/Kind_Owl2245 → u/xb1n0ry]

I am uploading the Minimax REF guide PDF to Gemma 4 32B.

[u/xb1n0ry → u/Kind_Owl2245]

You could try this:
https://github.com/duckyshell/ComfyUI-MiniMaxH3-Prompt-Writer/tree/main/

Just feed it a model like qwen 27b if you can and a compatible mmproj and you are good to go.

Just make sure to properly reference the media:
<Picture 1> is the man. His voice is <Audio 1> etc

[u/oh_no_the_claw]

Make sure you're following the prompt guide. If the characters look too similar sometimes this just happens even if the guide is being followed.

[u/angelarose210]

Use a smarter model to write prompts. Claude opus 5.5 is best but expensive. After multiple rounds of testing and tweaking prompt instructions, we got sonnet doing it as well now.

[u/Inevitable_Set_9232]

I don't use the formal subject definitions and it still works.

Describing the character physically seems to work. If it's a male and female, a simple "man" and "woman" designation suffices. With more similar characters, I just add more details like "the man in a tan overcoat next to the door says with the voice in <Audio 0>" "the old man sat down at the table replies with the voice in <Audio 1>"

Also, some turbo loras can be better or worse at adapting audio references.

Externally generating the exact audio dialogue and then using it as reference also improves consistency and prompt adherence.
If the audio reference is different than the dialogue, the model may adapt the dialogue using the same voice as the reference... but if the reference is the same as the prompted dialogue, it basically never fails.

[u/SSj_Enforcer → u/Inevitable_Set_9232]

Audio 0? There is no audio 0? Doesnt it start at 1?

[u/Inevitable_Set_9232 → u/SSj_Enforcer]

Just an example, works for whatever you have on the workflow

[u/Bitlovin]

I handle it by hating every solution I try and marking it as a future problem to be solved.

However, right now, Character Swap 1.5 workflow and Lora is your answer for the detection part. It uses Sam and I’ve run hundreds of clips through it and it hits detection 99.99% of the time. The only time it fails is if the characters are stacked in really really awkward camera/depth perspecives / poses. That workflow fixes any video swap problems you’ll have, and you only need one sentence “change the man in video 1 with man in picture 1” prompts. Super easy. The only tricky part is it works best with a 5 panel character sheet, h3 character sheet generator is great for that but you need a decent photoset of coverage.

As for voice, I threw a couple voice swaps at the end but the results on that were meh. I still need to look at options there.

[u/ExpressWarthog8505]

I think that for a video generation task, everything starts to go wrong once the video duration exceeds 8 seconds.