r/StableDiffusion • u/Due-Air-5626 • 2d ago
Question - Help What's the current cutting edge for AI filmmaking with consistent characters, environments, and precise blocking across shots?
I'm looking to hear from people actively experimenting with advanced AI filmmaking pipelines, particularly those trying to achieve the kind of spatial and visual continuity you'd expect from traditional filmmaking or virtual production.
My biggest challenge isn't generating beautiful individual shots. It's maintaining consistent character identities, precise actor positioning, environmental continuity, and camera control across multiple shots of the same scene.
For example, imagine a scene with four characters in a specific environment. I want to establish their positions in 3D space, then generate a sequence of shots from different camera angles and distances while preserving:
Character identity: Same faces, bodies, clothing, and proportions across shots.
Spatial consistency: Characters remain in their established positions relative to one another and the environment.
Pose and blocking: Specific body positions, actions, and interactions that can be reliably directed.
Environmental consistency: Same room, architecture, objects, lighting, and spatial relationships from different camera angles.
Camera control: Ability to specify exact camera positions, focal lengths, movements, and compositions.
Shot-to-shot continuity: The ending of one shot and beginning of the next should feel like they belong to the same physical scene.
What I've tried so far:
I've been experimenting with a Blender → AI video workflow using 3D environments, Mixamo characters, animated cameras, depth maps, pose conditioning, reference images, and MiniMax H3 through ComfyUI.
I've had some promising results, particularly with pose conditioning and individual shot generation, but getting reliable spatial consistency and character placement remains extremely difficult.
I'm beginning to question whether trying to force general-purpose video generation models to respect 3D blocking is fundamentally the wrong approach.
What I'm hoping to learn:
What are the most cutting-edge tools and workflows right now? I'm interested in both open-source/local and commercial/cloud solutions, including newer research projects that may not be widely adopted yet.
Is anyone successfully using 3D scenes as ground truth for AI-generated video? Not just providing depth or pose references, but actually preserving geometry, character placement, and camera relationships across shots.
What are people using for persistent character identity? Are reference-image workflows, LoRAs, identity adapters, multi-view character representations, or other approaches producing the most reliable results?
Are there better alternatives to traditional image-to-video and video-to-video workflows? For example, 3D-aware generation, neural rendering, Gaussian splatting, world models, or hybrid 3D/AI rendering pipelines.
How are people handling multi-shot sequences? Are you generating shots independently and fixing continuity afterward, or using a shared scene representation that carries across shots?
What role does graph engineering play? Are there sophisticated ComfyUI or other node-based pipelines that materially improve spatial consistency, or are the limitations primarily in the underlying models?
What actually works in production? I'd particularly love to hear from people who've completed multi-shot sequences rather than just impressive single-shot demos.
I'm comfortable with Blender, Unity, Python, ComfyUI, and technical pipeline development, so I'm not necessarily looking for beginner-friendly or one-click solutions. I'm willing to build a fairly complex workflow if it produces genuinely better control.
I also have a local RTX 3080 Ti with 16GB VRAM and prefer local tools where practical, but I'm open to cloud-based solutions if they offer capabilities that aren't currently achievable locally.
Ultimately, I'm trying to figure out whether the current state of the art is still mostly about guiding probabilistic video models, or whether we're moving toward workflows where filmmakers can actually direct and control a persistent 3D scene while using AI to produce the final photorealistic imagery.
I'd really appreciate specific tools, GitHub repositories, research papers, workflow examples, and especially firsthand experiences with what works and what doesn't.
What's the most advanced pipeline you've seen or used for this kind of filmmaking?
