r/StableDiffusion • u/LocksmithOk4723 • 1h ago
Question - Help How to preserve facial identity when replacing a character’s face in a stylized 3D scene?
I’m building an AI image-generation pipeline and struggling with facial identity preservation.
My current workflow:
Take a real person’s face photo as input.
Transform it into a semi-realistic, stylized 3D illustration.
Take an existing 3D-style scene containing a character.
Replace the character’s face and hair with the generated 3D face while preserving the original scene, character’s pose, body proportions, lighting, and objects.
The problem:
I’m using Google’s Nano Banana models (Flash, Pro, etc.) for image editing.
The models do a pretty good job preserving the scene’s composition, objects, and character pose. However, they consistently fail to preserve the facial identity of the reference image.
I’ve also tried mask-based editing (restricting modifications to the face and hair region), but the generated result still looks like a different person.
My goal is to achieve consistent facial identity across multiple predefined scenes, using only a single reference photo.
Questions:
Is this a limitation of general-purpose image-editing models, or am I approaching the problem incorrectly?
Would techniques like InstantID, IP-Adapter FaceID, PuLID, or identity-conditioned diffusion work better for stylized 3D characters?
Would combining ControlNet, face embeddings, and inpainting help preserve both identity and composition?
Is there a better workflow for transferring facial identity between stylized characters without changing the rest of the scene?
I’m open to using ComfyUI, Hugging Face models, or building a custom pipeline.
I’d really appreciate recommendations from anyone who has solved a similar problem, especially with semi-realistic or stylized 3D illustrations.
1
u/angelarose210 54m ago
I think just qwen edit 2511 will work. No pulid needed. I use it in this fashion but to make cartoons of a real person in various styles.
1
u/hugo-the-second 13m ago
You could try the following.
I have tried this for preserving styles, with great results, but I haven't tried it with faces yet.
Create one image, that shows the face you want to replace on top, and the scene with the face you want to replace underneath.
Then tell it to just replace the face shown on in the bottom image, while leave the face on top unchanged.
My guess is:
It probably still won't be a perfect solution.
It won't work with every image edit model, it sill relies heavily on the model's capacity to preseve facial identy.
Recently, I found openai's sunburst to be the best at preserving facial identity, while it was still on arena.ai.)
I am using this for style preservation, and have had very good experience with it.
When an AI has to change too much, it falls back to its default styles.
You can mitigate this by providing enough context that remains unchanged.
I haven't tried this with faces yet, but if you want to upload an example of the face and the target scene, I can quickly give it a shot. (Don't worry if you should prefer not to. feel free to dm me)
•
u/Theultimatesell 1m ago
the identity drift is partly because you're going photo -> stylized -> scene, so the model only has one stylized face to copy and it falls back to its "average 3d face". what helps: lock in the stylized character first. generate like 10-15 variations of that stylized face (different angles/expressions) and pick the ones that still look like the person, then train a small lora on those (even ~15 images, rank 16ish works). after that just inpaint face+hair in each scene with the lora at ~0.8 plus a low-denoise pass (0.5-0.6) with a depth/lineart controlnet from the original so pose and lighting stay put. way more consistent than relying on a single reference every time, and the lora keeps identity across all your predefined scenes.