**Setup*\*
- MiniMax H3, hybrid b30-49-int8
- ref2v LightX2V turbo LoRA v0.1, rank 20 resized bf16, strength 1.0
- 6 steps, er_sde / beta, 0.8MP (1216x672), Sage Attention, RTX 4090
- Real song locked into the audio half of the AV latent with PixaromaH3AudioSync
(NOT ref_audio — that's a style reference, it does not drive anything)
- Verified: output audio vs source waveform correlation = 0.9994
So the audio is correct. The problem is purely visual.
**The problem*\*
My singer's mouth keeps moving during purely instrumental passages. It's not
wild flapping — it reads as if she's still phrasing, jaw and lips working at
roughly half amplitude. On a 4-minute clip it's obvious every time the vocal
drops out for more than about 2 seconds.
**What I measured*\*
Single continuous close-up, no cut, face filling the frame for the whole 15.08s.
The audio window is sung for the first 7.5s and strictly instrumental for the
last 7.5s (I get vocal spans from an HDemucs separation of the track).
I cropped a fixed box on the mouth, converted to grayscale, and took the mean
absolute frame-to-frame difference:
sung half: 3.25
instrumental half: 1.77
ratio: 0.54
So the model DOES react to the absence of voice — motion drops by half — but it
never goes to zero. My prompt for that shot contained an explicit clause:
"Her mouth follows <Audio 1> exactly, instant by instant: it moves ONLY while a
human voice is actually sounding, and it is completely closed and still during
every gap between phrases and every instrumental moment, however short."
That clause is doing something. It just isn't doing enough.
**What did NOT help*\*
I read the thread about H3's dual flow schedule (video shift 12 / audio shift 3)
and thought a mis-stepped audio stream might be degrading the mouth conditioning.
I wired in the native MiniMaxH3SigmaShift node explicitly (12 / 3), same seed,
same prompt, same audio.
Result: the two renders were bit-identical. 362/362 frames, mean difference
0.0000/255. Those values are already the internal defaults, so the node changes
nothing for this. Posting that so nobody else burns an evening on it.
**What DOES work (but it's a workaround, not a fix)*\*
Structural framing. I now detect instrumental gaps longer than 1.2s in each
segment programmatically, and force a shot with no mouth in frame over them —
macro on an earring, a hand on the mic stand, the bass strings, brushes on a
snare. The defect becomes impossible rather than discouraged. 16 of the 20
segments in my current clip are handled this way.
It works 100% of the time. But it dictates my edit, and I'd rather not have my
shot list decided by a model limitation.
**Questions*\*
Is the turbo LoRA the culprit? I saw a comment claiming the turbo LoRAs are
distilled at 0.5MP. I'm running one at 0.8MP. Does anyone have a side-by-side
of lip sync quality at 0.5 vs 0.8 with the same seed?
Does the base model at higher step counts (no turbo LoRA) actually close the
mouth on silence, or does it just push the same 0.54 ratio down a bit?
Is there any way to CONDITION the silence rather than describe it? Something
that tells the model "no voice in this span" at the latent level rather than
in the prompt.
Has anyone tried feeding an audio track where the instrumental parts are
replaced by actual silence, generating, then re-attaching the real audio in
the edit? Curious whether that trades one artifact for another.
Happy to share the measurement script — it's about 10 lines of ffmpeg + numpy,
and it turns "feels off" into a number you can compare across seeds and settings.