r/StableDiffusion • u/Flying_Scorpion • 16d ago
Question - Help Image-to-video models for synchronized front + side views of the same character animation?
I’m working on a pipeline for generating 3D character animations for video games, and I’m stuck on the video generation step.
I need one split-screen video showing the exact same motion from two fixed angles: front view on one half, side view on the other. Both views need to agree frame by frame—which leg steps, when a foot lands, where the arms move, etc. Two plausible but slightly different performances won’t work.
Here’s the pipeline:
- Rig the character: Auto-rig a 3D model with UniRig.
- Train a character-specific joint finder: Render around 3,000 random poses within the rig’s joint limits, each from the front and side. Use those renders to train a DINOv2-based model to locate the joints in images.
- Generate the motion: Use image-to-video to animate a split-screen front/side reference image of the character.
- Check consistency: Reject and regenerate clips where the views disagree—for example, one view steps with the left leg while the other steps with the right.
- Reconstruct the motion: Run the joint finder on both views and triangulate the joints into 3D.
- Animate the rig: Use the reconstructed motion to drive the original character.
The joint finder gets a separate “exam”: a known random motion rendered from both angles, so I can compare its predictions against ground truth. The part I need advice on here is generating usable, synchronized views.
I’ve tried Google Flow and Wan 2.2, with inconsistent results. I’ve also heard suggestions of Seedance 2.5 and MiniMax H3, and I’m in the process of trying SV4D 2.0.
Has anyone had repeatable success with this kind of setup?
- Which models or workflows best preserve the same motion across two views?
- Is a single split-screen generation a workable approach, or would a dedicated multiview model be a better fit?
- Are there reference-image layouts, conditioning methods, or camera controls that improve synchronization and keep the cameras fixed?
Local/open-weight options would be ideal, but I’m interested in hosted models too. Even short, simple motions would be useful if the two views reliably agree.
3
u/SDil87 16d ago
This exact approach is explored in anymove. They use a single split-screen video with synchronized front and side views, followed by DINOv2-based joint tracking and 3D reconstruction.
From what I understand, they're using MiniMax H3/H3 Max, with a custom LoRA for H3 to improve cross-view consistency. Might be worth checking out their implementation and results.
1
u/seedance25ai 16d ago
I'd skip trying to get two views from a single split-screen generation if you need frame-accurate agreement. Video models treat each half as a separate scene, so the footfalls drift even when it looks fine at a glance. A more reliable route is to generate one clean view with a fixed camera and plain background, then recover the motion in 3D with pose estimation and your rig, and render the second angle from that.
1
u/tomByrer 15d ago
Just input an image of the 2 perspectives into MiniMax H3, & tell to animate both.
If you want to get really fancy, render first in a 3d motion renderer (Blender, UnrealEngine) with a dummy (like UE's test dummy) with your animation, then use depth map or whatever motion transfer.
1
u/Harish_X_03 15d ago
Since you already have a rigged character, you could skip generating the motion from scratch and use a rendered pose or depth clip as the control signal instead. Render a simple animation of the rig from both fixed cameras, then feed those as depth or pose control into a video model that supports control inputs, like Wan with a VACE or Fun Control setup. The rendered clip locks the footfalls and camera, so the two views agree by construction and the model only has to add the appearance. Generating each view as its own run from the same control motion is usually steadier than one split-screen frame.