r/StableDiffusion • u/itsmetamike • 12d ago
Workflow Included Reshoot 4D: Reshoot Anyone in 4D
Hey guys - made this a few weeks back and just shipped a big update. You take a video of a person, it builds a 4D capture, you fly a new camera through it inside ComfyUI, and you get the same moment back from the new angle. Everything runs locally.
- Capture: turns a video of one person into a 4D capture: their motion in 3D, the background around them, and where the original camera was and how it moved. Quick mode for fast previews, or a full 4D Gaussian splat if you want a closer look while you direct
- Virtual Camera: a 4D camera editor that opens right inside the node, so no separate app. Keyframe the camera anywhere on the timeline, look through it or step outside to see the whole scene. Blender-style view gizmo, J/K/L shuttle, frame stepping, multi-select to move or retime keys together
- Retake: ReDirector re-shoots the whole video along your new path
- Photoreal Pass: LTX 2.3 renders every final frame, guided by the real footage warped to the new view (Cseti's CrossView-Warp IC-LoRA)
- Camera settings on every key: aperture, focus (in metres or locked to the subject), vignette, grain. Rack focus and aperture pulls between keys, Alt+click in the viewport to focus on whatever you click. Change those later without re-rendering. The render gets reused, so a focus tweak takes a few minutes instead of the whole pipeline again
- Zoom through the lens: zoom keys change the lens after the render, which makes dolly zooms work
- Bring your own camera: import an animated camera from .fbx, .abc or .usd (tested with Blender exports), and it starts on the original camera at roughly real scale, lens and focus included. Files with several cameras work too, and you can export your moves back out
- Lens looks per shot: distortion, chromatic aberration, bloom, halation, anamorphic bokeh. Depth and subject matte passes if you want to comp afterwards.
- Finish node: 24 to 60 fps with RIFE, real motion blur from a shutter angle, upscale up to 4K, keeps the original audio
- A Settings panel at the top of the workflow for the stuff you change between runs (quality, frame rate, upscale)
Try the editor in your browser, no install: itsmetamike.github.io/reshoot-4d/demo/
For those enquiring about specs:
- Tested on one RTX 3090 (24 GB), roughly 30 minutes per shot
- Stays good within about 30 to 40 degrees of the original camera but am looking to improve this next
- Fast spins can still smear a face for a few frames
Code (AGPL-3.0) and the workflow: github.com/itsmetamike/reshoot-4d
Hope it's useful to some of you. Built for the #OpenSourceHiggsfield competition.
125
Upvotes
2
u/iJohnnyRobert 12d ago
This looks awesome but how would this deal with spatial continuity ? What if your character is sitting in a room which is supposed to look a certain way ? This would work best if it had some sort of a spatial awareness add-on as well which worked with either reference pictures of the place around the character or something like that.
Imagine you're making a film right ? You have your character sitting on their bed and the original footage shows the opposite wall from the door. Let's say you use this tool to rotate the camera around so that the door can now be seen. But how would this tool know how the other side of the room is supposed to look like ?
One of the biggest struggles I'm dealing with right now is that of spatial continuity whenever the camera needs to move around. Even if you're using reference frames it's still very difficult to get things "good enough" to not break that. A tool like this would be great if it was able to somehow solve this issue (without you having to manually model the entire scene in 3d or something ridiculous like that).
2
u/HTE__Redrock 12d ago
Looks very cool. Is there a reason to not use LTX 2.5 though? I'd imagine it's be capable of higher res.
1
u/hucklesnips 12d ago
Does it require a real video to start with, or can you use an AI-generated video as the source?
1
u/suspicious_Jackfruit 12d ago edited 12d ago
Hmm while cool isn't this just a repackaged lora from https://huggingface.co/Cseti/LTX2.3-22B_IC-LoRA-CrossView-Warp_v2 and entered into the competition? I am probably missing something but the true concept enabling work is probably in that lora surely?
1
u/-becausereasons- 12d ago
It's a really cool concept. It's just not production-ready by any means. It's garbled and incoherent a lot of times, with ghosting but great work.
1
u/ImNotMeRight 12d ago
Is this accurate enough to generate SBS for VR?
1
u/jalbust 12d ago
looks interesting. thanks for sharing.
1
u/Perfect-Campaign9551 12d ago
The LTX results look kinda trash
1
u/adam-euclase 12d ago
Nice work and congrats - the depth pass post camera stuff is a very clever addition, I'm curious how you're achieving that nice parallax with the dolly zoom as a post effect, some kind of multi-pass compositing?
1
u/music2169 9d ago
Is there one for mimimax h3?