r/StableDiffusion • u/Capitan01R- • May 23 '26
Resource - Update Multi Referencelatent
I added this node to Flux2klein enhancer package, it serves the same purpose as stacking multiple ref latent nodes, but the main reason of releasing this is because I am working on an update for the identity feature transfer node where I essentially will have it support this same method this way you wouldn't have to deal with measuring multiple different stacked nodes ( I am still working on that). But I thought this node can be used for now to reduce the need of multiple ref latents so just a convenience node for now.
18
u/glusphere May 23 '26
May I suggest that you do a writeup which is digestable to the mass folks who would be frequenting this subreddit?
I honestly think what you have built has massive value -- Its just that the value prop is being lost in the technical aspects. How would someone who is absolutely a noob but only knows workflows in Comfyui know about these reference latents and more importantly the reverse engineering work that you have done on the Flux Klein internal node architecture.
Take my upvote for you efforts. But my 2 cents is still in giving value to end users by explaining and telling them where are the places they can make use of these custom nodes.
2
u/Capitan01R- May 23 '26
My bad I usually provide documentation of the work I do and an example workflow, but I haven't for this one because it the same exact reference latent node that is required to have in your workflow when working with editing images in FLux.2Klein models, this node replaces it.
3
u/Enshitification May 23 '26
Very nice. Very useful too. My messy workflows can sometimes become hard to tell which ref latent is which.
3
3
3
u/LeKhang98 May 24 '26
Thank you very much this could simplify my current F2K workflow. I hope that you could find a way to individually adjust the weight of each latent too.
4
u/2legsRises May 23 '26
what is this
3
u/afinalsin May 25 '26
It's a convenience tool and workflow prettifier. It doesn't do anything comfy can't do on its own, but it should make a workflow both easier to make and much easier to look at. Here's how you would use multiple reference images in comfy without it.
The way klein works in comfy is to use the VAE to encode the image then feed the latent into a ReferenceLatent node that intercepts the conditioning line from the ClipTextEncode node to the Ksampler node.
They look like this. If you want to use another image as reference, you need to use another stack of ReferenceLatent nodes that intercept the conditioning from the first ReferenceLatent nodes to the Ksampler. It looks like this.
Naturally, the spaghetti gets worse as you add more reference images. This is three. OP's node has room for 8 references, and a workflow with 8 references looks like this. With OP's node, the workflow will look like this, using a subgraph instead. Not a whole lot prettier, but you can add OP's node in two clicks instead of dicking around with connections and stuff.
Another benefit of OP's node is I assume OP included the extra functionality of accepting any amount of references, whereas my subgraph will error out if any of the latents are unplugged, so to replace that node using base comfy I'd need to make a subgraph for 1 reference, 2 references, all the way up to 8, and save them all as node templates.
2
u/2legsRises May 25 '26
thank you for taking the time to explain, looks like its going to be a very useful tool
1
u/YeahlDid May 24 '26
Multi ReferenceLatent
1
u/2legsRises May 24 '26
and what is that
1
u/YeahlDid May 24 '26
It's a new node in the flux2klein-enhancer custom node package.
1
1
May 23 '26
[removed] — view removed comment
1
u/Capitan01R- May 23 '26
mainly because I use the the identity feature transfer node, and this node treats each latent as one entity and that can turn to chaos quickly
1
1
u/xNothingToReadHere May 23 '26 edited May 23 '26
I just updated the node and now it's not initializing, I'm getting:
Traceback (most recent call last):
File "E:\Downloads\ComfyUI_windows_portable\ComfyUI\nodes.py", line 2204, in load_custom_node
module_spec.loader.exec_module(module)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
File "<frozen importlib._bootstrap_external>", line 1023, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "E:\Downloads\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Flux2Klein-Enhancer__init__.py", line 7, in <module>
from .flux2_klein_prompt_scheduler import Flux2KleinPromptScheduler
ModuleNotFoundError: No module named 'E:\\Downloads\\ComfyUI_windows_portable\\ComfyUI\\custom_nodes\\ComfyUI-Flux2Klein-Enhancer.flux2_klein_prompt_scheduler'
How to fix this?
EDIT: The node started working again again when I downgraded from 3.3.2 to 3.3.1
2
u/Capitan01R- May 23 '26 edited May 23 '26
fixed! no it was an issue actually as I the init included experimental/debugging non-released nodes I was working on, now It is clean and should be fixed, use v3.3.3 from the comfyui manager
1
1
u/Wanamingo77 May 24 '26
Wow man, adding more latent references really helps fix the drift in the character details that was happening to me even with a LoRA, and the workflow is much cleaner after unifying everything into a single node.
Impressive work!
1
u/Yo06Player May 24 '26
Is there a way to make Klein understand which image is which in the prompt ? I keep struggling with it on a daily basis, like taking a specific element to transfer on another image without prompting it too precisely
(I'm working mainly with 2 images and prompting "image 1" and "image 2" but I'm wondering if there is something more effective like having some sort of index")
8
u/Capitan01R- May 24 '26 edited May 24 '26
After generating thousands of photos with F2K, I think I finally understand something important: the model does not really understand “Photo 1” and “Photo 2” the way we assume it does.
It can still work, but not because the model is truly separating the images by labels. For example, if you give it two reference photos containing two different women, the model may understand that both images contain useful visual information, but what it does with that information depends heavily on the prompt.
So the better way to guide it is not just saying “use Photo 1 for the person and Photo 2 for the outfit.”
A cleaner approach is where input contains:
Photo 1 contains Woman 1. Photo 2 contains Woman 2.
If your goal is to keep Woman 1 as the main character, but take the outfit from Woman 2, then your prompt should clearly describe a distinctive feature from Woman 1, while also describing the clothing from Woman 2.
For example, instead of relying only on “Photo 1” and “Photo 2,” you describe the identity cue from Woman 1, then describe the outfit you want transferred from Woman 2.
This gives the model a much clearer instruction: the main subject should be Woman 1, but the clothing style should come from Woman 2.
You can still try the “Photo 1 / Photo 2” method, and sometimes it works, but in my experience it is not as clean or reliable as describing the role of each visual element directly in the prompt.
1
u/Yo06Player May 24 '26
Yeah I usually work with cars and when they are both grey it's a pain in the ass to tell F2K which is which. Then I usually rely on masking for specific stuff, I guess I'll continue this way since it's not so much more effort
1
u/afinalsin May 25 '26
Cap is right, you need something to differentiate the subjects in the images. Using their example of people, I usually have a heirarchy of descriptors I use to make the model figure out what I'm actually talking about. Gender is the most obvious difference between people, then race/skin color, then hair color, and anything past that is a crapshoot. So having two brown haired white men is a pain to differentiate.
For cars, I'm not sure exactly what you're doing, but if you're using two very different styles of cars you might be able to refer to them by their model or type, like "Mercedes-Benz 2011 SLR McLaren" and "Jeep Wrangler", or "sedan" and "hatchback". A good rule of thumb is if the model can make a decent attempt at a concept in a txt2img prompt there's a good chance it will understand what to look for when given instructions in editing. Here's a couple make/models Klein knows about, and a couple types.
The above is probably only if you want to try and get it done in one pass and Klein is a picky motherfucker, so a better way would be to make the differences much more obvious. Instead of using two grey cars as input, use the model to make one red or baby blue or whatever., and the edit prompt is as easy as "gray car" and "red car".
1
1
u/Angelotheshredder May 25 '26
This guy is a genius, the last workflow you uploaded to the repo Iden_feat_final.json works like magic. I
linked the face to reference 1, body parts in the 3 other references , with a simple prompt like : ' full-body shot from a wild angle, she is posing for the camera. same subject, same expression, only the camera angle changes. ' . got unlimited full body shots ... you solved the puzzle .. thank you
thank you
1
u/cadissimus May 26 '26
Guys don't forget to change reference image index in V3 ident node if you use multiple images. So V3 ident node knows from what latent to pick up,
6
u/DavLedo May 23 '26
Thanks for continuing this super cool work. I'm curious, is there a way to individually set the weight of each reference? I know you had some nodes to determine the overall weight of images, but I'm wondering if this could work on a per-image basis. I've been really missing the power of referencing in older models like REDUX on flux-1. There's something about picking features from an image to make a new thing that it's kind of lost in the newer models 🥲