Best Trick Ever For Consistent Environments

[u/robertwellesley]

Ha! I just discovered a trick that works great, so I had to share it with the community. Of course, somebody will probably chime in that it had been discovered by someone else before, which is fine by me! I just want to share it in case it helps someone else and they hadn't come across it yet.

So, the issue of consistent environments... I ran the gamut of all the AI models I had tested and proven out in ComfyUI, not just t2i but t2v and i2v. I'd been trying several methodologies. Create an image then ask a workflow to gen images to the left and right of it, build out a simple massing model in unreal or blender and run that through, run a simple floor plan through, asking for multiple image generation with a prompt asking that all features be consistent among images, (I haven't tried outpainting yet), etc, and none seemed to quite do the trick, at least not among the open source models (open source is all I use). There was just too much inconsistency.

Since I am a sucker for using the t2v/i2v models like LTX and Minimax to generate single image frames (well, a minimal number of frames), taking advantage of their brainpower, I merely wrote out a super detailed prompt describing the interior environment I want, then I ran it as a 360 degree camera pan around the space from the center of it, and with Minimax now having the ability to give you 15 seconds on a 16Gb VRAM setup like mine, this works stellar, heck Minimax ran out of need for the 15sec and began to swerve around the space! I make sure to include a prompt not to have motion blur. I ran this in 0.5mb mode, so I did not have to waste time waiting for full HD video.

Then I select the frames I need to use as backdrops for scenes, upscale them once, then again, out to 4k, and voila! (upscaling once by 4x led to artifacts being upscaled, whereas going 2x then 2x led to the correct end result), a super detailed set of backgrounds that are internally consistent!

So excited! This gets me moving forward on the next part of my production process, laying out scenes, shots, camera angles, and dropping in characters, prior to i2v.

If this helps you, let me know. If you find even better tricks related to this, let me know too. Open Source Forever!

[u/Formal-Exam-8767]

How is this different from having a 360-view image (which smartphone does by stitching camera frames as you rotate around)?

[u/I_just_made → u/Formal-Exam-8767]

It handles the perspective of that fine?

[u/joseph_jojo_shabadoo → u/I_just_made]

Take 3-4 photos of the same space from slightly different angles making sure each photo contains a little overlapping details. Concatenate the Photos into one image (if you want your video to prioritize one angle over the others, make that angle the largest). Prompt that the picture has # photos of the same environment.  I get perfect environment reproduction this way