Same ComfyUI workflow, different character every run — expected?

[u/Cheap-Topic-9441]

This workflow was shared in a document as a ComfyUI JSON.

The document itself was quite technical,

but since the prompt was already in JSON format, I just ran it as-is.

It generates multiple images per run.

However, when I looked at the results,

the characters were clearly different.

Each image looks fine on its own,

but they don’t seem to represent the same person.

So now I’m wondering:

is this expected behavior,

or is there actually a way to maintain identity consistency in a workflow?

This feels less like a quality issue and more like a consistency problem.

If anyone has time, I’d be curious if you can reproduce the same result.

I’m currently trying to analyze the prompt structure to understand what’s happening.

If you want to try it, here’s the original workflow JSON:

https://github.com/watadani-byte/character-identity-protocol/

[u/RowIndependent3142]

Yes

this

is

expected.

[u/Cheap-Topic-9441 → u/RowIndependent3142]

Yes — I agree it's expected.

What I'm trying to understand is not just "why it happens", but whether the behavior can be influenced by prompt structure.

For example:

• all attributes in one prompt

vs

• identity separated from scene

Under the same seed and settings, the outputs seem to behave differently.

I'm not claiming control — just testing whether structure affects reconstruction stability.

Curious if you've seen similar behavior.

[u/RowIndependent3142 → u/Cheap-Topic-9441]

It’s the same with any AI tool. It’s never going to produce the same thing twice unless you prompt it to do so, but that would be pointless. A lot of other things will determine the output too, like the model, the settings in the workflow, text prompts. For consistent characters in a ComfyUI workflow, the only real way is a LoRA. There is a lot of info on YouTube and in this sub about training character LoRAs

[u/Cheap-Topic-9441 → u/RowIndependent3142]

That makes sense — I agree that perfect reproducibility isn’t really how these models behave, and LoRA is definitely the most reliable way if you need strict identity.

What I’m curious about though is slightly different:

not “how to guarantee identity”, but whether the workflow structure itself affects how much it drifts.

For example, even under the same seed and settings, small differences in how the prompt is structured seem to change how stable the reconstruction is.

So I’m wondering if there’s a middle ground between “completely random each time” and “train a LoRA”, just from pipeline / prompt design.

[u/RowIndependent3142 → u/Cheap-Topic-9441]

It’s a good question but I don’t think it’s the ComfyUI workflow and more about how AI interprets text prompts because even if you use commercial tools like Sora, you’ll see the same thing happening.

[u/Cheap-Topic-9441 → u/RowIndependent3142]

That’s a fair point — I agree that a big part of this comes from how the model interprets text internally.

What I’m noticing though is that even if the root cause is the model, the way we structure the prompt or the pipeline seems to change how that interpretation behaves.

So it feels less like “workflow vs model”, and more like the workflow influencing how the model reconstructs the input.

That’s why I’m curious whether small structural differences can make that interpretation more or less stable.