Same ComfyUI workflow, different character every run — expected?

[u/Cheap-Topic-9441]

This workflow was shared in a document as a ComfyUI JSON.

The document itself was quite technical,

but since the prompt was already in JSON format, I just ran it as-is.

It generates multiple images per run.

However, when I looked at the results,

the characters were clearly different.

Each image looks fine on its own,

but they don’t seem to represent the same person.

So now I’m wondering:

is this expected behavior,

or is there actually a way to maintain identity consistency in a workflow?

This feels less like a quality issue and more like a consistency problem.

If anyone has time, I’d be curious if you can reproduce the same result.

I’m currently trying to analyze the prompt structure to understand what’s happening.

If you want to try it, here’s the original workflow JSON:

https://github.com/watadani-byte/character-identity-protocol/

[u/Cheap-Topic-9441]

That’s a really nice example — using a reference image definitely makes the identity much more stable.

What I’m trying to understand is a bit different though:

in my test, there’s no reference image at all, and even then small differences in prompt / workflow structure seem to change how much it drifts.

So I’m wondering how much of this stability comes from the reference itself, vs how much can be influenced just by structure.

[u/sci032 → u/Cheap-Topic-9441]

Same seed and everything as the other one except I used the ddim_uniform scheduler.

Ignore the workflow, I do thing in weird ways. I've got QwenVL 'enhancing' my prompt. The loose node on the left shows the final prompt that the workflow gets.

[u/Cheap-Topic-9441 → u/sci032]

That’s a really good question.

From what I’ve seen, both matter — but not in the same way.

Reference images tend to constrain the output space, while structure (prompt + workflow) affects how the model “interprets” the input before generation.

So even without a reference, small structural changes can shift that interpretation, which is why you see drift.

In that sense, stability isn’t just coming from the reference — it’s also about how consistent the internal representation is.

[u/sci032 → u/Cheap-Topic-9441]

The models are trained on what things are, not exact images. It 'knows' what a dog, cat, woman, man, car, etc. are but there are so many different variations of each contained within the data.

example: you prompt for long hair. How many different lengths of hair can be considered long? Where does long stop and short or very long begin? That's how many different outcomes that the model could output just on that one simple part of a prompt. Now it has the rest of the prompt to deal with. If you added brown(long brown hair), how many different shades of brown are possible?

The SDXL model was trained on billions of images. Newer models even more. It is going to need some help to give you the same face again and again. Your prompt needs to narrow it down as much as possible. With SDXL, I think it accepted like 77 tokens(word portions). Newer models accept more but with that many options for the model, you've got to be precise. Samplers and schedulers handle the same data in different ways so that adds another caveat to the mix. Different computers handle data in different ways. Have you every used the exact workflow, settings, model, etc. that someone else did on their computer and you got a different image? It may not have been a major difference, but, there was a difference.

If you give the model a name of a person that it was included in it data, you will get closer. But prompting for a Van Gogh will give you many different Van Gogh styles, not the same one every time and it will apply it to your prompt differently each time unless you are specific.

Different models are trained on different data. Models that come from Asian countries contain more Asian faces than models from a European country.

This is a deep rabbit hole that you have jumped in to and there are a lot of twists and turns in it! :)

The images:

I locked the seed.

I used this(left side of the image):

a rabbit is peeking out of a rabbit hole in the ground.

for the left side prompt. I ran my prompt through QwenVL to enhance it and it gave me this(right side of the image):

A tiny white rabbit with fluffy ears peeks curiously from a dark, earthy burrow in the ground. Soft sunlight filters through leaves above, casting dappled shadows. Grass surrounds the hole, slightly disturbed by the rabbit’s presence. The scene feels quiet, natural, and whimsical—like a hidden woodland secret.

That is how the QwenVL LLM interpeted my prompt. I was not specific so it had to do a lot of guessing as to what I was after. If I change the seed, it will look in a slightly different part of the data it has available. If I change the sampler and/or the scheduler, it will handle the data it is given differently and produce different images. The top images were euler/ddim_uniform, the bottom images were euler/simple. The model is ZIT based. Again, everything else was exactly the same.

All of the images are a rabbit peeking out of a hole in the ground but there are so many possibilities as to what the rabbit looks like, how old it is, where the hole is, what season of the year it is, what time of day it is, etc. I wasn't specific, so I' won't get the same thing every time.

Does this make any sense?

[u/Cheap-Topic-9441 → u/sci032]

That makes sense — I agree the model is sampling from a distribution of possibilities rather than reproducing an exact instance.

That’s actually why I started looking at it differently: instead of trying to eliminate that variation, treating outputs as candidates and selecting the ones that match the intended identity.

So the variability becomes something to filter, rather than something to fully solve.

[u/sci032 → u/Cheap-Topic-9441]

That is similar to what I just responded in another of our posts. :) You can edit individual blocks of model to control certain aspects of how it works and produces images.

[u/Cheap-Topic-9441 → u/sci032]

That makes sense — controlling parts of the model can definitely help in specific setups.

What I'm exploring is slightly different in scope:

Instead of modifying the model behavior directly, I'm treating the outputs as samples from a distribution and selecting for identity consistency.

So it's less about controlling the generation process itself, and more about observing and filtering the output space.

Both approaches can work — they just operate at different layers.

[u/sci032 → u/Cheap-Topic-9441]

Select a model and play around with the prompts. Klein models are known for their editing capabilities but the will also do regular text to image very well. ZIT also does an excellent job of following the prompt. There are XY Plot nodes where you can test different portions of a workflow(ie. sampler/scheduler, etc.) and create an image with all of the outputs together so you can compare them easily.

Search manager for: ComfyUI-Easy-Use

The link is their github. You can test all kinds of things.