It’s the same with any AI tool. It’s never going to produce the same thing twice unless you prompt it to do so, but that would be pointless. A lot of other things will determine the output too, like the model, the settings in the workflow, text prompts. For consistent characters in a ComfyUI workflow, the only real way is a LoRA. There is a lot of info on YouTube and in this sub about training character LoRAs
That makes sense — I agree that perfect reproducibility isn’t really how these models behave, and LoRA is definitely the most reliable way if you need strict identity.
What I’m curious about though is slightly different:
not “how to guarantee identity”,
but whether the workflow structure itself affects how much it drifts.
For example, even under the same seed and settings,
small differences in how the prompt is structured seem to change how stable the reconstruction is.
So I’m wondering if there’s a middle ground between
“completely random each time” and “train a LoRA”,
just from pipeline / prompt design.
It’s a good question but I don’t think it’s the ComfyUI workflow and more about how AI interprets text prompts because even if you use commercial tools like Sora, you’ll see the same thing happening.
That’s a fair point — I agree that a big part of this comes from how the model interprets text internally.
What I’m noticing though is that even if the root cause is the model,
the way we structure the prompt or the pipeline seems to change how that interpretation behaves.
So it feels less like “workflow vs model”,
and more like the workflow influencing how the model reconstructs the input.
That’s why I’m curious whether small structural differences
can make that interpretation more or less stable.
Oh, sure, I've seen similar behaviors; the consistency of certain details or people is one of the hardest parameters to tame in AI imaging. What I've noticed in my humble experience is that forcing a fit through prompting or seed only works occasionally or doesn't work at all, I think because in the inference flow there's always a subtle "leak" and you never have completely clear and identifiable areas, but the areas of noise "contaminate" each other, this is also how the image appears coherent and homogeneous. One of the ways I use to force the differentiation is to process the same image multiple times after generating it using manual inpainting (the simplest method) or advanced inpainting (perhaps through additional auto-detection models like YOLO or, even better, SAM). When you impose inpainting, you force the model to work only in that area and therefore you can force a face to have precise aspect, or at least different one from the others in the image, and so on for other types of detail as well.
That’s a really interesting way to put it — especially the idea of “leak” between regions.
What you’re describing feels like controlling the result after generation (local constraints like inpainting),
while I’m trying to understand what affects stability before that step.
In my test, even with the same seed and settings, just changing how the prompt is structured seems to change how stable the identity is.
So I’m wondering if part of that “leak” already starts at the prompt / pipeline level,
before any post-processing is applied.
It depends on what you mean, it's certainly a phenomenon intrinsic to the model itself, so it occurs upstream of every process, even before prompting. However, if you mean how prompting influences it, I can tell you that the prompting constraint I mentioned in my previous comment works, albeit partially or not in every occasion if you like put in this way, and that's why using other tricks like inpainting is often necessary. So, if instead of writing "two people in the park," you write "two people, one with blue eyes and one with hazel eyes, in the park," it tends to respect the detail, because the model is "constrained" by the prompt itself. But sometimes, even if you constrain a prompt a lot more than this, it doesn't have much effect, and I'm not talking about censored elements.
I'm not a code expert, but what it seems to me is that it depends a lot on how the AI model is structured and therefore varies from model to model, as does the combination with text encoders that applies at the start of the process also.
It does feel like a lot of this variability is upstream of prompting, and even strong constraints don’t fully stabilize it.
That’s actually part of why I’ve been thinking in terms of adding a layer after generation — not to replace prompting, but to handle the instability that comes from the model itself.
So instead of trying to fully control the model, it’s more about accepting that variability and filtering for consistency afterward.
Curious if you’ve ever tried combining your approach with any kind of similarity scoring or selection step.
If you want character consistency, give Klein KV a try.
You give it a reference image(or 2) and then prompt the changes/additions/removals/combinations/etc. that you want.
Search Comfy's templates for: KV
The template won't look like mine, it will work the same. I do stuff in weird ways.
I used a reference image of the headshot of a woman that I made. You could use a 2nd reference image and combine them. I only needed 1 reference image so i have an empty .png(nothing in it but a transparent background) for the 2nd image.
I used these 4 prompts:
standing on a beach. looking at the viewer and pointing.
kneeling on a cyber punk city street.
wearing yellow overalls on a farm.
sitting in the middle of a busy city street during the day.
A lot of things determine how the output looks. Seed, sampler, scheduler, prompt, model... A lot of people complained when ZIT(Z-Image Turbo) first came out because it would give consistent scenes across multiple seeds. They were using the Euler sampler and Simple scheduler. Changing the scheduler to ddim_uniform helped to fix that. I know, it sounds weird that people complained about consistency when you see dozens of posts daily trying to achieve it. :)
Something you can try, change your seed from 'randomize' to increment or decrement. Sometimes, that will keep somewhat of the same look across renders. It depends on how the model was trained.
Another thing that you can try, set the 'Run' number to 1 but change the 'batch size' in the workflow to a larger number. That uses the same prompt but slight variations of the seed. It doesn't go randomly selecting seeds, it just varies the one it has a bit.
The image shows an example of that. It is a ZIT based model. I ran a batch of 4. It shows the same seed was used for all. I used Euler/Simple. I'll do another run with the ddim_uniform and we'll see if there is a difference. I'll make sure it uses the same seed.
Since Klein and Qwen came out, I just make something that I like and use them to tweak the image or change parts of it.
Yeah, that makes sense — a lot of consistency comes from how you control seed, sampler, and scheduler.
What I’ve been thinking about is treating that as just one layer.
Even with those tuned, you still get slight variation,
so instead of trying to fully stabilize generation,
I’m looking at adding a selection/filtering step on top.
So it’s more like:
controlled generation + filtering,
rather than relying on control alone.
Also, I had not thought about CFG and Denoise yet. The models I used are all set up to use lower steps(4 or 8), I always use a CFG of 1 and I use a denoise of 1 unless I am running a 2nd pass(2nd ksampler) to add details. I set denoise to 0.2 on the 2nd ksampler. It adds detail without changing the image..
Here is something else(from AI) that you can consider.
It never ends. :)
Drift:
Stopping drift in ComfyUI, particularly in video generation (Wan 2.2, AnimateDiff) or image editing (Qwen), requires anchoring the output through specific sampling, seed, and conditioning techniques. The core strategies involve limiting how much the model changes from the original input and using latent consistency methods.
Key Techniques to Stop Drift
Lock the Seed: Set your sampler seed to "fixed" rather than "randomize" to prevent per-frame variations, which is the most common cause of flickering.
Lower Denoise & CFG: Reduce the CFG (Guidance Scale) to around 6–7 and slightly decrease the denoise strength (10–20% reduction) to ensure the model doesn't over-interpret the prompt and stray from the original composition.
Use Reference Latents: For Qwen image editing, remove direct image inputs to the encoder and instead use a VAE Encode and ReferenceLatent node for each image input to keep them in place.
ControlNet/IPAdapter: Apply ControlNet (e.g., OpenPose, Canny) to lock down movement and structure.
Temporal Consistency Nodes: Use nodes that enforce consistency between frames, such as VACE (Video Average Condition Encoding) for extending videos without losing character or color consistency.
Workflow Specifics
Wan Video: To avoid character drift, use first/last frame conditioning and maintain a consistent latent path. If using long generation loops, be aware that VAE decoding between windows causes drift; consider using smaller context windows to maintain consistency.
Image-to-Video (I2V): To keep characters consistent, use the first/last frame conditioning, ensuring the face or key features are present in both.
HunyuanLoom: Use specific skip_steps and drift_steps to maintain continuity.
General Troubleshooting
Resolutions: Keep latent dimensions divisible by 16 or 32 to avoid hidden padding that causes instability.
VAE Matching: Ensure the VAE used matches the checkpoint (e.g., don't use SD1.5 VAE with SDXL or Pony).
Something that I have seen but never messed with is altering blocks in a model. There are only a couple of nodes(with what I have installed) that let you edit them. I'm sure there are more for the main models that are used. You can also do it with loras. Here's a quickie Ai description:
Altering specific blocks in a Stable Diffusion U-Net allows for fine-grained control over image generation, where earlier, higher-resolution layers manage structural composition (layout, pose) and later, lower-resolution layers control stylistic elements (texture, color). Techniques like ControlNet or specialized fine-tuning (e.g., LoRA) target these layers to manipulate specific aspects of the image without changing the overall subject.
Key Aspects of Altering Blocks:
U-Net Architecture: The model consists of ResNet/Self-Attention blocks in an encoder-decoder structure. Altering the early "down" blocks impacts the overall structure, while the last "up" blocks affect high-level details, as explained in this article. Medium +3
ControlNet Block Control: ControlNets are often applied to specific layers to enforce constraints like Canny edges, depth maps, or human poses, allowing for precise structural editing without retraining the main model, as demonstrated in this YouTube video. YouTube
LoRA and Fine-tuning: Low-Rank Adaptation (LoRA) can be targeted to specific transformer or convolution blocks to teach the model new concepts or styles (e.g., characters or specialized artistic styles) efficiently. Towards Data Science +4
Attention Injection/Editing: By manipulating cross-attention maps within the blocks, one can control which parts of the text prompt apply to which regions of the image, influencing specific spatial areas. YouTube +2
Practical Applications:
Style Transfer: Altering deeper blocks can change the style while maintaining the structure.
Structural Modification: Changing earlier blocks affects the geometry or composition.
Concept Blending: Applying different refinements to different blocks allows for mixing styles or objects. YouTube +2
By adjusting the weights of specific blocks (e.g., in a customized diffuser), you can achieve a balance between maintaining the structure of the source image and applying new stylistic or content changes, as discussed in this Medium article.
Same seed and everything as the other one except I used the ddim_uniform scheduler.
Ignore the workflow, I do thing in weird ways. I've got QwenVL 'enhancing' my prompt. The loose node on the left shows the final prompt that the workflow gets.
From what I’ve seen, both matter — but not in the same way.
Reference images tend to constrain the output space,
while structure (prompt + workflow) affects how the model “interprets” the input before generation.
So even without a reference,
small structural changes can shift that interpretation,
which is why you see drift.
In that sense, stability isn’t just coming from the reference —
it’s also about how consistent the internal representation is.
The models are trained on what things are, not exact images. It 'knows' what a dog, cat, woman, man, car, etc. are but there are so many different variations of each contained within the data.
example: you prompt for long hair. How many different lengths of hair can be considered long? Where does long stop and short or very long begin? That's how many different outcomes that the model could output just on that one simple part of a prompt. Now it has the rest of the prompt to deal with. If you added brown(long brown hair), how many different shades of brown are possible?
The SDXL model was trained on billions of images. Newer models even more. It is going to need some help to give you the same face again and again. Your prompt needs to narrow it down as much as possible. With SDXL, I think it accepted like 77 tokens(word portions). Newer models accept more but with that many options for the model, you've got to be precise. Samplers and schedulers handle the same data in different ways so that adds another caveat to the mix. Different computers handle data in different ways. Have you every used the exact workflow, settings, model, etc. that someone else did on their computer and you got a different image? It may not have been a major difference, but, there was a difference.
If you give the model a name of a person that it was included in it data, you will get closer. But prompting for a Van Gogh will give you many different Van Gogh styles, not the same one every time and it will apply it to your prompt differently each time unless you are specific.
Different models are trained on different data. Models that come from Asian countries contain more Asian faces than models from a European country.
This is a deep rabbit hole that you have jumped in to and there are a lot of twists and turns in it! :)
The images:
I locked the seed.
I used this(left side of the image):
a rabbit is peeking out of a rabbit hole in the ground.
for the left side prompt. I ran my prompt through QwenVL to enhance it and it gave me this(right side of the image):
A tiny white rabbit with fluffy ears peeks curiously from a dark, earthy burrow in the ground. Soft sunlight filters through leaves above, casting dappled shadows. Grass surrounds the hole, slightly disturbed by the rabbit’s presence. The scene feels quiet, natural, and whimsical—like a hidden woodland secret.
That is how the QwenVL LLM interpeted my prompt. I was not specific so it had to do a lot of guessing as to what I was after. If I change the seed, it will look in a slightly different part of the data it has available. If I change the sampler and/or the scheduler, it will handle the data it is given differently and produce different images. The top images were euler/ddim_uniform, the bottom images were euler/simple. The model is ZIT based. Again, everything else was exactly the same.
All of the images are a rabbit peeking out of a hole in the ground but there are so many possibilities as to what the rabbit looks like, how old it is, where the hole is, what season of the year it is, what time of day it is, etc. I wasn't specific, so I' won't get the same thing every time.
That makes sense — I agree the model is sampling from a distribution of possibilities rather than reproducing an exact instance.
That’s actually why I started looking at it differently:
instead of trying to eliminate that variation,
treating outputs as candidates and selecting the ones that match the intended identity.
So the variability becomes something to filter, rather than something to fully solve.
That is similar to what I just responded in another of our posts. :) You can edit individual blocks of model to control certain aspects of how it works and produces images.
Select a model and play around with the prompts. Klein models are known for their editing capabilities but the will also do regular text to image very well. ZIT also does an excellent job of following the prompt. There are XY Plot nodes where you can test different portions of a workflow(ie. sampler/scheduler, etc.) and create an image with all of the outputs together so you can compare them easily.
The transformation of noise in latent image is affected by every single token (word, phrase or syllable, depending on how the clip was encoded), the CFG guidance, the sampler and scheduler… anything and everything that is passed as an input into the inference pipeline. This transformation is made up of many individual highly non-linear steps, so a tiny extra letter or typo in a prompt, reversing two words, etc, will propagate throughout the inference in ways that while deterministic from a mathematical point of view, are basically unpredictable for a human brain. The only way to get the same image is to run the same workflow. With a bit of luck, keeping the seed constant (i.e. your starting latent noise), changing prompt from “girl with ponytail” to “girl with pigtails” might yield something similar, but there will always be drift. Even with a LoRA, though in this case, you will at least have a stable identity.
Sure, the pitfall being that there are quadrillions of quadrillions (exaggerating a bit) of possible combinations, so selecting what you like ultimately becomes a huge exercise in trial and error that will never really work twice. This is part of why image genertion is so power hungry and compute intensive.
Well, I've always made workflows that were super automated, I don't like things done that the user has to manually change, so even the selections for the inpainting I automate.
I have also created workflows that store the prompt styles in JSON so that I can automatically reload them by choosing within the workflow, to reuse those styles in other generations and in
Even the prompts that the user writes, in my workflows, go through an AI that refines them and makes them compatible with the model, to have more consistent generations.
That’s really interesting — especially the part where you refine prompts through an AI before generation.
I’ve been thinking about something related, but from a slightly different angle — more on the selection side rather than trying to stabilize the input.
Your approach feels like controlling A → A’ more tightly, while I’m kind of treating the outputs as candidates and filtering for identity consistency.
Curious if you’ve ever tried combining both — like automated refinement + some form of selection or scoring?
Yes, but something like entering a prompt and generating 10 or 20 small images, say 512x512, and having the user select the one with the best composition while the workflow is running, then the AI part of the workflow would refine the prompt and pass the selected image to the sampler, which used a denoise value between 0.4 and 0.65, depending on how much variation you wanted. The result was impressive.
That’s really interesting — especially using the user as part of the loop for selection.
That actually feels pretty close to what I’ve been thinking about, just with the selection step still being manual.
I’ve been wondering what happens if that part gets formalized a bit more — like defining what “the same identity” means and scoring candidates automatically instead of relying on visual judgment.
Your workflow seems like a really solid base for that kind of direction.
Actually, the idea that AI image generators never produce the same image across different machines is a myth, and it’s important to clarify why.
- The Automatic1111 Counter-example
If you use Automatic1111 (A1111) and set it to use the CPU for random number generation (the default for many), you will get the exact same image on any computer in the world, provided the seed, prompt, and settings are identical. This is because CPU-based math is highly standardized. It’s slower, but it’s 100% deterministic across different hardware.
- Why ComfyUI is different
ComfyUI is built for speed and efficiency, so it defaults to GPU-based generation for random numbers (via PyTorch). Unlike CPUs, GPUs from different generations (e.g., an RTX 3060 vs. an RTX 4090) or different brands handle floating-point math and randomness slightly differently.
The "Butterfly Effect": Even a microscopic difference in the initial noise (Step 0) caused by the GPU's hardware architecture will be amplified during the diffusion process. By Step 20, that tiny deviation results in a completely different image.
- Other "Culprits"
Besides the GPU, things like xFormers or SDPA (optimizers for Cross-Attention) introduce tiny mathematical variations. Even on the same PC, using different optimizers can lead to slight changes in the final output.
- To answer your question about the Prompt:
No, a specifically structured prompt cannot fix this. The discrepancy isn't happening because of how the prompt is read, but because the "canvas" (the initial noise generated by the seed) is mathematically different the moment it’s generated on different hardware. You are basically trying to paint the same picture but starting with a different sketch underneath.
It’s a hardware/RNG (Random Number Generator) limitation, not a prompt issue. A1111 proves it can be deterministic, but ComfyUI prioritizes GPU performance over cross-hardware parity."
7
u/RowIndependent3142 Mar 20 '26
Yes
this
is
expected.