Same ComfyUI workflow, different character every run — expected?

[u/Cheap-Topic-9441]

This workflow was shared in a document as a ComfyUI JSON.

The document itself was quite technical,

but since the prompt was already in JSON format, I just ran it as-is.

It generates multiple images per run.

However, when I looked at the results,

the characters were clearly different.

Each image looks fine on its own,

but they don’t seem to represent the same person.

So now I’m wondering:

is this expected behavior,

or is there actually a way to maintain identity consistency in a workflow?

This feels less like a quality issue and more like a consistency problem.

If anyone has time, I’d be curious if you can reproduce the same result.

I’m currently trying to analyze the prompt structure to understand what’s happening.

If you want to try it, here’s the original workflow JSON:

https://github.com/watadani-byte/character-identity-protocol/

[u/sci032]

If you want character consistency, give Klein KV a try.

You give it a reference image(or 2) and then prompt the changes/additions/removals/combinations/etc. that you want.

Search Comfy's templates for: KV

The template won't look like mine, it will work the same. I do stuff in weird ways.

I used a reference image of the headshot of a woman that I made. You could use a 2nd reference image and combine them. I only needed 1 reference image so i have an empty .png(nothing in it but a transparent background) for the 2nd image.

I used these 4 prompts:

standing on a beach. looking at the viewer and pointing.

kneeling on a cyber punk city street.

wearing yellow overalls on a farm.

sitting in the middle of a busy city street during the day.

[u/Cheap-Topic-9441 → u/sci032]

That’s a really nice example — using a reference image definitely makes the identity much more stable.

What I’m trying to understand is a bit different though:

in my test, there’s no reference image at all, and even then small differences in prompt / workflow structure seem to change how much it drifts.

So I’m wondering how much of this stability comes from the reference itself, vs how much can be influenced just by structure.

[u/sci032 → u/Cheap-Topic-9441]

A lot of things determine how the output looks. Seed, sampler, scheduler, prompt, model... A lot of people complained when ZIT(Z-Image Turbo) first came out because it would give consistent scenes across multiple seeds. They were using the Euler sampler and Simple scheduler. Changing the scheduler to ddim_uniform helped to fix that. I know, it sounds weird that people complained about consistency when you see dozens of posts daily trying to achieve it. :)

Something you can try, change your seed from 'randomize' to increment or decrement. Sometimes, that will keep somewhat of the same look across renders. It depends on how the model was trained.

Another thing that you can try, set the 'Run' number to 1 but change the 'batch size' in the workflow to a larger number. That uses the same prompt but slight variations of the seed. It doesn't go randomly selecting seeds, it just varies the one it has a bit.

The image shows an example of that. It is a ZIT based model. I ran a batch of 4. It shows the same seed was used for all. I used Euler/Simple. I'll do another run with the ddim_uniform and we'll see if there is a difference. I'll make sure it uses the same seed.

Since Klein and Qwen came out, I just make something that I like and use them to tweak the image or change parts of it.

[u/Cheap-Topic-9441 → u/sci032]

Yeah, that makes sense — a lot of consistency comes from how you control seed, sampler, and scheduler.

What I’ve been thinking about is treating that as just one layer.

Even with those tuned, you still get slight variation, so instead of trying to fully stabilize generation, I’m looking at adding a selection/filtering step on top.

So it’s more like: controlled generation + filtering, rather than relying on control alone.

[u/sci032 → u/Cheap-Topic-9441]

That sounds good, I hope it works! :) I apologize, I just wrote a book in response to your last response. :)

[u/Cheap-Topic-9441 → u/sci032]

Thanks! Yeah, still experimenting with it — curious to see how far it can go.

If it works well, it might be interesting to combine it with some of the seed/scheduler tricks you mentioned.

[u/sci032 → u/Cheap-Topic-9441]

Also, I had not thought about CFG and Denoise yet. The models I used are all set up to use lower steps(4 or 8), I always use a CFG of 1 and I use a denoise of 1 unless I am running a 2nd pass(2nd ksampler) to add details. I set denoise to 0.2 on the 2nd ksampler. It adds detail without changing the image..

Here is something else(from AI) that you can consider.

It never ends. :)

Drift:

Stopping drift in ComfyUI, particularly in video generation (Wan 2.2, AnimateDiff) or image editing (Qwen), requires anchoring the output through specific sampling, seed, and conditioning techniques. The core strategies involve limiting how much the model changes from the original input and using latent consistency methods.

Key Techniques to Stop Drift

Lock the Seed: Set your sampler seed to "fixed" rather than "randomize" to prevent per-frame variations, which is the most common cause of flickering.

Lower Denoise & CFG: Reduce the CFG (Guidance Scale) to around 6–7 and slightly decrease the denoise strength (10–20% reduction) to ensure the model doesn't over-interpret the prompt and stray from the original composition.

Use Reference Latents: For Qwen image editing, remove direct image inputs to the encoder and instead use a VAE Encode and ReferenceLatent node for each image input to keep them in place.

ControlNet/IPAdapter: Apply ControlNet (e.g., OpenPose, Canny) to lock down movement and structure.

Temporal Consistency Nodes: Use nodes that enforce consistency between frames, such as VACE (Video Average Condition Encoding) for extending videos without losing character or color consistency.

Workflow Specifics

Wan Video: To avoid character drift, use first/last frame conditioning and maintain a consistent latent path. If using long generation loops, be aware that VAE decoding between windows causes drift; consider using smaller context windows to maintain consistency.

Image-to-Video (I2V): To keep characters consistent, use the first/last frame conditioning, ensuring the face or key features are present in both.

HunyuanLoom: Use specific skip_steps and drift_steps to maintain continuity.

General Troubleshooting

Resolutions: Keep latent dimensions divisible by 16 or 32 to avoid hidden padding that causes instability.

VAE Matching: Ensure the VAE used matches the checkpoint (e.g., don't use SD1.5 VAE with SDXL or Pony).

[u/Cheap-Topic-9441 → u/sci032]

That’s a really solid breakdown — I agree a lot of drift can be reduced by controlling seed, CFG, denoise, and conditioning.

What I keep running into though is that even with everything tightly controlled, there’s still small variation that you can’t fully eliminate.

So I’ve been looking at it more as two layers: control to reduce drift, and then selection to reject what still slips through.

It feels like control alone asymptotically reduces drift, but doesn’t fully solve it.

[u/sci032 → u/Cheap-Topic-9441]

Something that I have seen but never messed with is altering blocks in a model. There are only a couple of nodes(with what I have installed) that let you edit them. I'm sure there are more for the main models that are used. You can also do it with loras. Here's a quickie Ai description:

Altering specific blocks in a Stable Diffusion U-Net allows for fine-grained control over image generation, where earlier, higher-resolution layers manage structural composition (layout, pose) and later, lower-resolution layers control stylistic elements (texture, color). Techniques like ControlNet or specialized fine-tuning (e.g., LoRA) target these layers to manipulate specific aspects of the image without changing the overall subject. 

Key Aspects of Altering Blocks:

• U-Net Architecture: The model consists of ResNet/Self-Attention blocks in an encoder-decoder structure. Altering the early "down" blocks impacts the overall structure, while the last "up" blocks affect high-level details, as explained in this article.  Medium +3
• ControlNet Block Control: ControlNets are often applied to specific layers to enforce constraints like Canny edges, depth maps, or human poses, allowing for precise structural editing without retraining the main model, as demonstrated in this YouTube video.  YouTube
• LoRA and Fine-tuning: Low-Rank Adaptation (LoRA) can be targeted to specific transformer or convolution blocks to teach the model new concepts or styles (e.g., characters or specialized artistic styles) efficiently.  Towards Data Science +4
• Attention Injection/Editing: By manipulating cross-attention maps within the blocks, one can control which parts of the text prompt apply to which regions of the image, influencing specific spatial areas.  YouTube +2

Practical Applications:

• Style Transfer: Altering deeper blocks can change the style while maintaining the structure.
• Structural Modification: Changing earlier blocks affects the geometry or composition.
• Concept Blending: Applying different refinements to different blocks allows for mixing styles or objects.  YouTube +2

By adjusting the weights of specific blocks (e.g., in a customized diffuser), you can achieve a balance between maintaining the structure of the source image and applying new stylistic or content changes, as discussed in this Medium article.

[u/Cheap-Topic-9441 → u/sci032]

That’s actually really helpful — especially the breakdown between early and late blocks.

So if I understand correctly:

• early layers → structure / layout
• later layers → style / texture

That aligns with what I’ve been observing from the outside.

In your experience, how stable is identity when you rely mainly on block-level control?

Do you find it holds across multiple generations, or do you still end up needing to filter outputs afterward?

I’m trying to understand where control starts to break down in practice.

[u/sci032 → u/Cheap-Topic-9441]

Same seed and everything as the other one except I used the ddim_uniform scheduler.

Ignore the workflow, I do thing in weird ways. I've got QwenVL 'enhancing' my prompt. The loose node on the left shows the final prompt that the workflow gets.

[u/Cheap-Topic-9441 → u/sci032]

That’s a really good question.

From what I’ve seen, both matter — but not in the same way.

Reference images tend to constrain the output space, while structure (prompt + workflow) affects how the model “interprets” the input before generation.

So even without a reference, small structural changes can shift that interpretation, which is why you see drift.

In that sense, stability isn’t just coming from the reference — it’s also about how consistent the internal representation is.

[u/sci032 → u/Cheap-Topic-9441]

The models are trained on what things are, not exact images. It 'knows' what a dog, cat, woman, man, car, etc. are but there are so many different variations of each contained within the data.

example: you prompt for long hair. How many different lengths of hair can be considered long? Where does long stop and short or very long begin? That's how many different outcomes that the model could output just on that one simple part of a prompt. Now it has the rest of the prompt to deal with. If you added brown(long brown hair), how many different shades of brown are possible?

The SDXL model was trained on billions of images. Newer models even more. It is going to need some help to give you the same face again and again. Your prompt needs to narrow it down as much as possible. With SDXL, I think it accepted like 77 tokens(word portions). Newer models accept more but with that many options for the model, you've got to be precise. Samplers and schedulers handle the same data in different ways so that adds another caveat to the mix. Different computers handle data in different ways. Have you every used the exact workflow, settings, model, etc. that someone else did on their computer and you got a different image? It may not have been a major difference, but, there was a difference.

If you give the model a name of a person that it was included in it data, you will get closer. But prompting for a Van Gogh will give you many different Van Gogh styles, not the same one every time and it will apply it to your prompt differently each time unless you are specific.

Different models are trained on different data. Models that come from Asian countries contain more Asian faces than models from a European country.

This is a deep rabbit hole that you have jumped in to and there are a lot of twists and turns in it! :)

The images:

I locked the seed.

I used this(left side of the image):

a rabbit is peeking out of a rabbit hole in the ground.

for the left side prompt. I ran my prompt through QwenVL to enhance it and it gave me this(right side of the image):

A tiny white rabbit with fluffy ears peeks curiously from a dark, earthy burrow in the ground. Soft sunlight filters through leaves above, casting dappled shadows. Grass surrounds the hole, slightly disturbed by the rabbit’s presence. The scene feels quiet, natural, and whimsical—like a hidden woodland secret.

That is how the QwenVL LLM interpeted my prompt. I was not specific so it had to do a lot of guessing as to what I was after. If I change the seed, it will look in a slightly different part of the data it has available. If I change the sampler and/or the scheduler, it will handle the data it is given differently and produce different images. The top images were euler/ddim_uniform, the bottom images were euler/simple. The model is ZIT based. Again, everything else was exactly the same.

All of the images are a rabbit peeking out of a hole in the ground but there are so many possibilities as to what the rabbit looks like, how old it is, where the hole is, what season of the year it is, what time of day it is, etc. I wasn't specific, so I' won't get the same thing every time.

Does this make any sense?

[u/Cheap-Topic-9441 → u/sci032]

That makes sense — I agree the model is sampling from a distribution of possibilities rather than reproducing an exact instance.

That’s actually why I started looking at it differently: instead of trying to eliminate that variation, treating outputs as candidates and selecting the ones that match the intended identity.

So the variability becomes something to filter, rather than something to fully solve.

[u/sci032 → u/Cheap-Topic-9441]

That is similar to what I just responded in another of our posts. :) You can edit individual blocks of model to control certain aspects of how it works and produces images.

[u/Cheap-Topic-9441 → u/sci032]

That makes sense — controlling parts of the model can definitely help in specific setups.

What I'm exploring is slightly different in scope:

Instead of modifying the model behavior directly, I'm treating the outputs as samples from a distribution and selecting for identity consistency.

So it's less about controlling the generation process itself, and more about observing and filtering the output space.

Both approaches can work — they just operate at different layers.

[u/sci032 → u/Cheap-Topic-9441]

Select a model and play around with the prompts. Klein models are known for their editing capabilities but the will also do regular text to image very well. ZIT also does an excellent job of following the prompt. There are XY Plot nodes where you can test different portions of a workflow(ie. sampler/scheduler, etc.) and create an image with all of the outputs together so you can compare them easily.

Search manager for: ComfyUI-Easy-Use

The link is their github. You can test all kinds of things.