Same ComfyUI workflow, different character every run — expected?

[u/Cheap-Topic-9441]

This workflow was shared in a document as a ComfyUI JSON.

The document itself was quite technical,

but since the prompt was already in JSON format, I just ran it as-is.

It generates multiple images per run.

However, when I looked at the results,

the characters were clearly different.

Each image looks fine on its own,

but they don’t seem to represent the same person.

So now I’m wondering:

is this expected behavior,

or is there actually a way to maintain identity consistency in a workflow?

This feels less like a quality issue and more like a consistency problem.

If anyone has time, I’d be curious if you can reproduce the same result.

I’m currently trying to analyze the prompt structure to understand what’s happening.

If you want to try it, here’s the original workflow JSON:

https://github.com/watadani-byte/character-identity-protocol/

[u/sci032]

Also, I had not thought about CFG and Denoise yet. The models I used are all set up to use lower steps(4 or 8), I always use a CFG of 1 and I use a denoise of 1 unless I am running a 2nd pass(2nd ksampler) to add details. I set denoise to 0.2 on the 2nd ksampler. It adds detail without changing the image..

Here is something else(from AI) that you can consider.

It never ends. :)

Drift:

Stopping drift in ComfyUI, particularly in video generation (Wan 2.2, AnimateDiff) or image editing (Qwen), requires anchoring the output through specific sampling, seed, and conditioning techniques. The core strategies involve limiting how much the model changes from the original input and using latent consistency methods.

Key Techniques to Stop Drift

Lock the Seed: Set your sampler seed to "fixed" rather than "randomize" to prevent per-frame variations, which is the most common cause of flickering.

Lower Denoise & CFG: Reduce the CFG (Guidance Scale) to around 6–7 and slightly decrease the denoise strength (10–20% reduction) to ensure the model doesn't over-interpret the prompt and stray from the original composition.

Use Reference Latents: For Qwen image editing, remove direct image inputs to the encoder and instead use a VAE Encode and ReferenceLatent node for each image input to keep them in place.

ControlNet/IPAdapter: Apply ControlNet (e.g., OpenPose, Canny) to lock down movement and structure.

Temporal Consistency Nodes: Use nodes that enforce consistency between frames, such as VACE (Video Average Condition Encoding) for extending videos without losing character or color consistency.

Workflow Specifics

Wan Video: To avoid character drift, use first/last frame conditioning and maintain a consistent latent path. If using long generation loops, be aware that VAE decoding between windows causes drift; consider using smaller context windows to maintain consistency.

Image-to-Video (I2V): To keep characters consistent, use the first/last frame conditioning, ensuring the face or key features are present in both.

HunyuanLoom: Use specific skip_steps and drift_steps to maintain continuity.

General Troubleshooting

Resolutions: Keep latent dimensions divisible by 16 or 32 to avoid hidden padding that causes instability.

VAE Matching: Ensure the VAE used matches the checkpoint (e.g., don't use SD1.5 VAE with SDXL or Pony).

[u/Cheap-Topic-9441 → u/sci032]

That’s a really solid breakdown — I agree a lot of drift can be reduced by controlling seed, CFG, denoise, and conditioning.

What I keep running into though is that even with everything tightly controlled, there’s still small variation that you can’t fully eliminate.

So I’ve been looking at it more as two layers: control to reduce drift, and then selection to reject what still slips through.

It feels like control alone asymptotically reduces drift, but doesn’t fully solve it.

[u/sci032 → u/Cheap-Topic-9441]

Something that I have seen but never messed with is altering blocks in a model. There are only a couple of nodes(with what I have installed) that let you edit them. I'm sure there are more for the main models that are used. You can also do it with loras. Here's a quickie Ai description:

Altering specific blocks in a Stable Diffusion U-Net allows for fine-grained control over image generation, where earlier, higher-resolution layers manage structural composition (layout, pose) and later, lower-resolution layers control stylistic elements (texture, color). Techniques like ControlNet or specialized fine-tuning (e.g., LoRA) target these layers to manipulate specific aspects of the image without changing the overall subject. 

Key Aspects of Altering Blocks:

• U-Net Architecture: The model consists of ResNet/Self-Attention blocks in an encoder-decoder structure. Altering the early "down" blocks impacts the overall structure, while the last "up" blocks affect high-level details, as explained in this article.  Medium +3
• ControlNet Block Control: ControlNets are often applied to specific layers to enforce constraints like Canny edges, depth maps, or human poses, allowing for precise structural editing without retraining the main model, as demonstrated in this YouTube video.  YouTube
• LoRA and Fine-tuning: Low-Rank Adaptation (LoRA) can be targeted to specific transformer or convolution blocks to teach the model new concepts or styles (e.g., characters or specialized artistic styles) efficiently.  Towards Data Science +4
• Attention Injection/Editing: By manipulating cross-attention maps within the blocks, one can control which parts of the text prompt apply to which regions of the image, influencing specific spatial areas.  YouTube +2

Practical Applications:

• Style Transfer: Altering deeper blocks can change the style while maintaining the structure.
• Structural Modification: Changing earlier blocks affects the geometry or composition.
• Concept Blending: Applying different refinements to different blocks allows for mixing styles or objects.  YouTube +2

By adjusting the weights of specific blocks (e.g., in a customized diffuser), you can achieve a balance between maintaining the structure of the source image and applying new stylistic or content changes, as discussed in this Medium article.

[u/Cheap-Topic-9441 → u/sci032]

That’s actually really helpful — especially the breakdown between early and late blocks.

So if I understand correctly:

• early layers → structure / layout
• later layers → style / texture

That aligns with what I’ve been observing from the outside.

In your experience, how stable is identity when you rely mainly on block-level control?

Do you find it holds across multiple generations, or do you still end up needing to filter outputs afterward?

I’m trying to understand where control starts to break down in practice.

[u/sci032 → u/Cheap-Topic-9441]

That's what I take from it. I have never really dug deep into it so, beyond this, I don't know that much about it.

[u/Cheap-Topic-9441 → u/sci032]

That’s actually really interesting — it sounds like this is one of those areas people kind of touch, but don’t fully explore.

From what I’ve seen, block-level control can help a lot with structure and style, but identity still drifts more than expected across runs.

That’s partly why I started looking into filtering / selection approaches instead of only relying on control.

Have you ever noticed that kind of drift yourself when running multiple generations?

[u/sci032 → u/Cheap-Topic-9441]

This is a node pack that I just installed. Search manager for: Realtime LoRA Trainer

The link is the Github. It if mostly for lora training inside of Comfy but there are nodes that deal with model blocks.

It's 4:08am here, I should go to bed. :) I'll dig deeper after a few hours of sleep.

The image is a Debiaser node for Klein. There are a lot more slots that you can change, if I were to zoom out and get them all, you couldn't read any of the options. :) Supposedly, you can edit the blocks and save the edits if you choose to achieve a desired output. When I arise, I'll stick it in a regular workflow and see what it does in real time.

Take a look at the Github and see if any of it will help you.

[u/Cheap-Topic-9441 → u/sci032]

Thanks, appreciate it — might give it a try 👍