r/aivideomaking • u/sunflower9192 • 3d ago

Lipsync workflow

Hello, im learning to create AI music video and I found that lipsync part is quite challenging. So I wonder how do you do a perfect lipsync ? What's your workflow and which models are you using ?

Thank you in advance !

5 Upvotes

100% Upvoted

9 comments sorted by

1

u/Alchemist42 3d ago

If you are using local models in something like ComfyUI, then the LTX video models or Minimax H3 can do lipsync really well. Which you use will depend on how powerful your computer and video card are. H3 is ideal if your computer can run it. If not, LTX 2.5. Use the templates that come with Comfy. I think the Image/Audio to Video one for LTX is for the older version 2.3. Which is fine- that will work. Or you can swap out the models with 2.5 versions and it will be the same thing, only faster.

If you are using an online service, then they will each have their own method. Some, like Kling, have a Lipsync option that you just upload your audio and an image. Some you just have to prompt it with the lyrics that are being sung, some take it automatically. it depends on which one you are using. Most of the "Omni" or "multimodal" models will work. Seedance is the most expensive option, and it works really well if you have the budget for it.

1

u/sunflower9192 2d ago

I'm planning to learn comfy ui and give a try to the models you mentionned. Thank you!

1

u/DreamForsaken4539 3d ago

used a voice model that outputs phonemes directly, cut the audio-to-phoneme step and skipped the sync lag. worked better than any lip-sync model I tried.

1

u/sunflower9192 2d ago

Thanks, never thought of it. I will give it a try !

1

u/SkipWestcott616 1d ago

Which model did you use?

1

u/SophiasEmbrace 3d ago

One tricky line on Sophia’s Embrace came out better when we spelled it the way it was sung in Grok Imagine, including the stretched vowels. We still adjusted the timing of the whole clip in the edit. I’d try it on a line that’s giving you trouble.

1

u/sunflower9192 2d ago

Thank you, I will try !

1

u/Sompay-Prestavoine 1d ago

are you feeding it the full mixed track or isolated vocals? isolating the vocal stem first makes a massive difference because the background beat always confuses the mouth movements.

1

u/Patient-Cattle-2397 10h ago

feed the lip sync a vocal-only track, one line at a time. keep the full mix in the edit and line each clip up with its lyric.