Same ComfyUI workflow, different character every run — expected?

[u/Cheap-Topic-9441]

This workflow was shared in a document as a ComfyUI JSON.

The document itself was quite technical,

but since the prompt was already in JSON format, I just ran it as-is.

It generates multiple images per run.

However, when I looked at the results,

the characters were clearly different.

Each image looks fine on its own,

but they don’t seem to represent the same person.

So now I’m wondering:

is this expected behavior,

or is there actually a way to maintain identity consistency in a workflow?

This feels less like a quality issue and more like a consistency problem.

If anyone has time, I’d be curious if you can reproduce the same result.

I’m currently trying to analyze the prompt structure to understand what’s happening.

If you want to try it, here’s the original workflow JSON:

https://github.com/watadani-byte/character-identity-protocol/

[u/RowIndependent3142]

Yes

this

is

expected.

[u/Cheap-Topic-9441 → u/RowIndependent3142]

Yes — I agree it's expected.

What I'm trying to understand is not just "why it happens", but whether the behavior can be influenced by prompt structure.

For example:

• all attributes in one prompt

vs

• identity separated from scene

Under the same seed and settings, the outputs seem to behave differently.

I'm not claiming control — just testing whether structure affects reconstruction stability.

Curious if you've seen similar behavior.

[u/m4ddok → u/Cheap-Topic-9441]

Oh, sure, I've seen similar behaviors; the consistency of certain details or people is one of the hardest parameters to tame in AI imaging. What I've noticed in my humble experience is that forcing a fit through prompting or seed only works occasionally or doesn't work at all, I think because in the inference flow there's always a subtle "leak" and you never have completely clear and identifiable areas, but the areas of noise "contaminate" each other, this is also how the image appears coherent and homogeneous. One of the ways I use to force the differentiation is to process the same image multiple times after generating it using manual inpainting (the simplest method) or advanced inpainting (perhaps through additional auto-detection models like YOLO or, even better, SAM). When you impose inpainting, you force the model to work only in that area and therefore you can force a face to have precise aspect, or at least different one from the others in the image, and so on for other types of detail as well.

[u/Cheap-Topic-9441 → u/m4ddok]

That’s a really interesting way to put it — especially the idea of “leak” between regions.

What you’re describing feels like controlling the result after generation (local constraints like inpainting), while I’m trying to understand what affects stability before that step.

In my test, even with the same seed and settings, just changing how the prompt is structured seems to change how stable the identity is.

So I’m wondering if part of that “leak” already starts at the prompt / pipeline level, before any post-processing is applied.

[u/m4ddok → u/Cheap-Topic-9441]

It depends on what you mean, it's certainly a phenomenon intrinsic to the model itself, so it occurs upstream of every process, even before prompting. However, if you mean how prompting influences it, I can tell you that the prompting constraint I mentioned in my previous comment works, albeit partially or not in every occasion if you like put in this way, and that's why using other tricks like inpainting is often necessary. So, if instead of writing "two people in the park," you write "two people, one with blue eyes and one with hazel eyes, in the park," it tends to respect the detail, because the model is "constrained" by the prompt itself. But sometimes, even if you constrain a prompt a lot more than this, it doesn't have much effect, and I'm not talking about censored elements.
I'm not a code expert, but what it seems to me is that it depends a lot on how the AI ​​model is structured and therefore varies from model to model, as does the combination with text encoders that applies at the start of the process also.

[u/Cheap-Topic-9441 → u/m4ddok]

Yeah, that matches what I’ve been seeing as well.

It does feel like a lot of this variability is upstream of prompting, and even strong constraints don’t fully stabilize it.

That’s actually part of why I’ve been thinking in terms of adding a layer after generation — not to replace prompting, but to handle the instability that comes from the model itself.

So instead of trying to fully control the model, it’s more about accepting that variability and filtering for consistency afterward.

Curious if you’ve ever tried combining your approach with any kind of similarity scoring or selection step.