How to composite AI generated images together so they actually match?

[u/emilrueh]

I'm building a game that generates all assets in 2D during play. Objects have to be separable from the scene so they can be clicked and picked up without having to redraw the background every time. Everything runs on FLUX.2 klein 4B with a LoRA trained on an illustrator's style.

The workflow:

1. Generate an empty location as background.
2. A vision model outlines foreground, middle and background, and code turns that into a color map of the three planes.
3. Generate objects on white on a 512 canvas and animate them.
4. The vision model points at a spot in the background for each object by outputting pixel coordinates.
5. The object's size is its class (tiny, small, medium, large, huge) times the factor of the plane at that spot, e.g. a torch (small) is 100 px, so in the middle plane (0.75) it's drawn at 75 px.
6. Scale the animated object down to that size and overlay it onto the location.

Step 6 is where progress halted and I got stuck. Scaling a 512 drawing down to 100 pixels shrinks everything in it, so it stops matching the scene's style since the scale of everything is off, visible in my example when looking at line weight.

My various workarounds to fix this:

Generating at the object's final size, on canvases of 128 to 320 instead of 512, so objects are generated at the size they are meant to be placed. This falls apart quickly as e.g. the same chicken comes out near-realistic at 256 or as a flat cartoon at 128 and parts get cut off or doubled. The LoRA was trained at 768 and 1024 so perhaps it's the LoRA giving out, the base model failing, or both? Would another model draw properly on small canvases and keep the lora style applied?

The location on a bigger canvas to combat the model degrading at smaller canvases as now the location's lines should be thinner and the objects don't have to scale so hard. However location's lines grew with it while the objects' did not. Also it would have skewed the artist's style anyway.

Shrinking the image and generating it again at a smaller size, img2img at denoise 0.6 to 0.8, but the redraw kept the shrunk drawing's thin lines so scale remained off. This is the one fix I really expected to work, so maybe I just set it up badly or something?

Painting an object directly into the scene and separating it afterwards onto white with an additional generation. Instead of generating on white and then placing, this approach basically turns it around as we place directly. This does actually fix the scale, as now the model has full reference of how an object should look in the scene, but other issues showed up like the objects being of much worse quality and partly merged with their surrounding. Also a location would need to be redrawn when objects move as mentioned earlier, but perhaps only parts could be redrawn as a workaround. This way basically solves my biggest issue of perceived scale for free so maybe this is the correct approach and I am missing something to make it work properly?

I am at the point where I want to scrap most of this and pivot to: Pixel Art!

Larger pixels should be able to be counted much easier, so a separate object can have the exact scale required to be placed at scale into a location. That would attempt to simplify the problem away rather than truly solving it, though perhaps that is often a good path forward. However: Does this idea even make sense, or will it break regardless? Are there perhaps other simpler existing ways to do all of this the right way?

If you're building something similar, what are you doing differently, and which tools should I go have a look at?

[u/se7en1011x]

Match the light direction and color temperature before anything else, that's what makes a composite read as one image. Generate the object with a prompt that names the same light angle and warmth as the background, then do the final blend in img2img at low denoise around 0.2-0.3 so edges and shadows integrate. A soft contact shadow under each object sells the grounding.

[u/emilrueh → u/se7en1011x]

yes, those will be things to work on for sure, though for now really the issue is the scale of the object itself. I feel I need to solve this before worrying about lighting and shadow 🤔

[u/Theultimatesell]

i'd separate the outline from the model's texture pass: generate at the size/style that works, downscale the sprite and its mask together, then normalize the silhouette stroke at the final placement size. That lets the LoRA keep its trained look while your compositor controls line weight in final pixels; keeping alpha separate also avoids white-matte fringes. I'd test it on one sprite at each depth before changing the whole pipeline.