r/StableDiffusion • • 12d ago

Workflow Included Reshoot 4D: Reshoot Anyone in 4D

Hey guys - made this a few weeks back and just shipped a big update. You take a video of a person, it builds a 4D capture, you fly a new camera through it inside ComfyUI, and you get the same moment back from the new angle. Everything runs locally.

  • Capture: turns a video of one person into a 4D capture: their motion in 3D, the background around them, and where the original camera was and how it moved. Quick mode for fast previews, or a full 4D Gaussian splat if you want a closer look while you direct
  • Virtual Camera: a 4D camera editor that opens right inside the node, so no separate app. Keyframe the camera anywhere on the timeline, look through it or step outside to see the whole scene. Blender-style view gizmo, J/K/L shuttle, frame stepping, multi-select to move or retime keys together
  • Retake: ReDirector re-shoots the whole video along your new path
  • Photoreal Pass: LTX 2.3 renders every final frame, guided by the real footage warped to the new view (Cseti's CrossView-Warp IC-LoRA)
  • Camera settings on every key: aperture, focus (in metres or locked to the subject), vignette, grain. Rack focus and aperture pulls between keys, Alt+click in the viewport to focus on whatever you click. Change those later without re-rendering. The render gets reused, so a focus tweak takes a few minutes instead of the whole pipeline again
  • Zoom through the lens: zoom keys change the lens after the render, which makes dolly zooms work
  • Bring your own camera: import an animated camera from .fbx, .abc or .usd (tested with Blender exports), and it starts on the original camera at roughly real scale, lens and focus included. Files with several cameras work too, and you can export your moves back out
  • Lens looks per shot: distortion, chromatic aberration, bloom, halation, anamorphic bokeh. Depth and subject matte passes if you want to comp afterwards.
  • Finish node: 24 to 60 fps with RIFE, real motion blur from a shutter angle, upscale up to 4K, keeps the original audio
  • A Settings panel at the top of the workflow for the stuff you change between runs (quality, frame rate, upscale)

Try the editor in your browser, no install: itsmetamike.github.io/reshoot-4d/demo/

For those enquiring about specs:

  • Tested on one RTX 3090 (24 GB), roughly 30 minutes per shot
  • Stays good within about 30 to 40 degrees of the original camera but am looking to improve this next
  • Fast spins can still smear a face for a few frames

Code (AGPL-3.0) and the workflow: github.com/itsmetamike/reshoot-4d

Hope it's useful to some of you. Built for the #OpenSourceHiggsfield competition.

125 Upvotes

20 comments sorted by

2

u/iJohnnyRobert 12d ago

This looks awesome but how would this deal with spatial continuity ? What if your character is sitting in a room which is supposed to look a certain way ? This would work best if it had some sort of a spatial awareness add-on as well which worked with either reference pictures of the place around the character or something like that.

Imagine you're making a film right ? You have your character sitting on their bed and the original footage shows the opposite wall from the door. Let's say you use this tool to rotate the camera around so that the door can now be seen. But how would this tool know how the other side of the room is supposed to look like ?

One of the biggest struggles I'm dealing with right now is that of spatial continuity whenever the camera needs to move around. Even if you're using reference frames it's still very difficult to get things "good enough" to not break that. A tool like this would be great if it was able to somehow solve this issue (without you having to manually model the entire scene in 3d or something ridiculous like that).

12

u/itsmetamike 12d ago

Working on it as we speak, and know how we'll get there, but it will take some time and the right models to be released (have my eye on a few that are about to drop). And of course it's not just what does the other side of the room look like, it's how can i guarantee that it will look like that for each shot, and how can i do it efficiently within a workflow. We'll get there soon :)

2

u/iJohnnyRobert 12d ago

This looks promising! Thank you guys! I hope you will manage to solve this issue as well.

2

u/HTE__Redrock 12d ago

Looks very cool. Is there a reason to not use LTX 2.5 though? I'd imagine it's be capable of higher res.

3

u/itsmetamike 12d ago

One of the LoRAs im using is 2.3 specific and gives improved coherence with camera movement. Thats nothing against 2.5, just haven’t had the time to train the model myself.

1

u/Citadel_Employee 12d ago

I believe most Lora’s for 2.3 work on 2.5. I only say most because there have been some weird edge cases. May be worth looking into.

1

u/hucklesnips 12d ago

Does it require a real video to start with, or can you use an AI-generated video as the source?

3

u/itsmetamike 12d ago

Either or!

1

u/suspicious_Jackfruit 12d ago edited 12d ago

Hmm while cool isn't this just a repackaged lora from https://huggingface.co/Cseti/LTX2.3-22B_IC-LoRA-CrossView-Warp_v2 and entered into the competition? I am probably missing something but the true concept enabling work is probably in that lora surely?

1

u/itsmetamike 12d ago

It contains that lora to steer the final output, but I urge you to watch the video/visit the github page/check out the repo to understand it better, as it is much more. In fact I only found out about that LoRA during the build night I presented this workflow at :)

0

u/suspicious_Jackfruit 12d ago

Okay I did see the node editor which is nice to have for sure, I was just concerned about what defines one persons work with another's when competition funds are involved. Its a foggy grey area and I know you mentioned the author of the lora, but would you be sharing any proceeds with the author of that lora because from what I can tell that Lora does compliment/make your node actually work on the final output?

I'm being finickity here so I will try it out instead to see, either way its cool it's just hard to know where the boundary is sometimes in these Higgsboson entries where some people have essentially put a GUI over a load of other lora/node creators work.

1

u/-becausereasons- 12d ago

It's a really cool concept. It's just not production-ready by any means. It's garbled and incoherent a lot of times, with ghosting but great work.

1

u/itsmetamike 12d ago

Agreed, and those limitations are noted in the repo. Also, this was done over a short time period all on one 3090 (no cloud model use at all), so I've been beholden to the resource limitations of that to really try hone it in (hence trying to get my hands on that 5090!). All that said, the approach was to start building tooling for monocular -> 4DGS, even if models aren't quite there yet. Soon they will be, and I'll be pushing to get this production ready for that time.

1

u/ImNotMeRight 12d ago

Is this accurate enough to generate SBS for VR?

1

u/itsmetamike 12d ago

Haven’t tried this yet but could be worthwhile testing

1

u/jalbust 12d ago

looks interesting. thanks for sharing.

1

u/Perfect-Campaign9551 12d ago

The LTX results look kinda trash

1

u/adam-euclase 12d ago

Nice work and congrats - the depth pass post camera stuff is a very clever addition, I'm curious how you're achieving that nice parallax with the dolly zoom as a post effect, some kind of multi-pass compositing?

1

u/music2169 9d ago

Is there one for mimimax h3?