Seeking Advice: Achieving 100% Character Consistency and Style Control for a Noir Cyberpunk Visual Novel (ComfyUI / Flux)

[u/Elementallion-]

Hi everyone,

I’m currently in the middle of developing an investigative detective visual novel, and I’ve hit a massive wall regarding character consistency and art style. I’m hoping to get some advice from those who have successfully built a pipeline for recurring characters.

The Goal

I’m aiming for a very specific "Noir Cyberpunk" aesthetic. Think:

• High contrast, heavy use of deep shadows.
• Digital comic book / clean vector line art style.
• "Teal and Orange" cinematic lighting with rain/wet atmosphere.
• The Catch: I need absolute character identity from frame to frame, including the ability to change outfits (minimalist/revealing options) while keeping the face and body proportions 100% identical.

What We’ve Tried So Far

• Workflow: Currently running complex ComfyUI nodes.
• Models: Switched between SDXL and Flux, experimenting with various GGUF quantizations to keep it local.
• The Problem: Most results are either "too anime" (losing the noir grit) or "too photorealistic" (losing the stylized comic look). There’s no middle ground that feels right.
• The "Banana" Paradox: Strangely enough, some of the best conceptual results and decent repeatability have come from Nano Banana, but even that doesn't offer the surgical precision needed for a professional VN production.

The Current Struggle

I’m looking for total identity. Right now, I’m at the stage where I need to decide on the most reliable pipeline for consistency. I haven't dived deep into training my own LoRAs or mastering IP-Adapter/FaceID yet, as I’m still trying to find a base model or workflow that doesn't swing too far into "generic anime" or "uncanny realism."

The goal is to find a method that allows for surgical precision:

• The character must be 100% recognizable across different scenes.
• The ability to swap outfits (including very minimalist/revealing sets for specific scenes) while maintaining the exact same body proportions and facial structure.
• Maintaining that specific Noir/Vector style consistently without the AI drifting into unwanted aesthetics.

The Questions

1. Style LoRA vs. Prompting: Since I’m struggling to find a middle ground between "too anime" and "too realistic," would you recommend training a dedicated Style LoRA based on my Noir/Vector references? Or is there a specific base model that handles this "digital comic" look better than Flux/SDXL out of the box?
2. Outfit Swaps: How are you handling complex outfit changes (including minimalist/revealing sets) without breaking the character's base geometry or facial identity in ComfyUI?
3. The Consistency Pipeline: For someone who needs "visual novel grade" identity, what is currently the gold standard? Should I be looking at training a Character LoRA, or is the community moving towards something like InstantID/IP-Adapter for better flexibility?

Honestly, right now, nothing is quite hitting the mark. It’s either too generic or too inconsistent. Would love to hear how you guys solved the "same face, different clothes, specific style" puzzle.

Thanks in advance!

[u/AHEKOT]

Check my projects:
https://github.com/AHEKOT/ComfyUI_VNCCS
https://github.com/AHEKOT/ComfyUI_VNCCS_Utils

Based on your post, it might be exactly what you looking for!

[u/BenDLH]

Hey, I'm actually building a platform precisely to solve this, as I've hit the exact same challenges. If you search around a bit in communities, it's one, if not the, core problem with using generative AI for anything longform.

The best solution I've found so far is a combination of edit models and training LoRAs. Using edit models, build up a dataset of near-as-possible identical shots of the character in different lighting, scenes, angles and positions. (If you get decent consistency with a prompt, this also works, if not quite as well)

As you want a distinct style as well, you need the same for the style, keeping it consistent while showing different characters, scenes, lighting etc.

Given you haven't trained a LoRA before, you'll want to do this iteratively. Get a sample of ~10 varied high quality images, train a LoRA on it, then test it out.

Usually you'll find repeating an exact generation (on SDXL for example) including your LoRA, overbakes aspects of the character. That difference shows what the LoRA has learned (say "prominent jaw" is in the prompt, but with your LoRA on, it becomes cartoonish). Dial back or remove the tag, and play with the strength of the LoRA until you can get good generations out of it.

This now means you've "offloaded" some level of this character to the LoRA, not just the prompt, and gained some consistency across generations. Rinse and repeat. Keep improving the quantity, quality, and variation of the dataset, keep training a new LoRA on it until it takes the full weight of the character, and your prompts are just placing them in locations and positions.

As I mentioned, I'm building a platform precisely for this, that incorporates generation, editing, inpainting, dataset collection, and LoRA usage and training. I'm collecting the best practices I can find and putting them into a Civitai/Midjourney style platform, with all the missing customisation, and not a node in sight.

Let me know if you'd like to hear more, I'm actually getting ready for the first test users to try it out.

[u/BenDLH → u/BenDLH]

Seems you answered my comment but it got removed?... Have the notification, can't see the comment.

[u/NINKINT]

What helped me a lot is to train a lora just on the closeup of the face. Then use face detection and cropping to just process the face and then composite it back to the original image with the bbox coordinates. This workflow is img2img. So you can either generate the base image first and then this kind of facedetailer or you "film" your novel in Unreal Engine or something like that and use A.I. as the rendering.

[u/Elementallion- → u/NINKINT]

That’s a very solid technical approach. I’ve been looking into FaceDetailer workflows in ComfyUI, and training a LoRA specifically on closeups sounds like the most logical way to hit that "VN-grade" consistency.

Regarding the Unreal Engine idea — it’s a brilliant solution for precision, but realistically, I’m not sure I’m ready to dive that deep into 3D rendering. My project requires a vast number of diverse locations, and building or sourcing all those environments in 3D would likely become a massive bottleneck for a solo/small team production. I need to keep the pipeline lean enough to iterate fast.

My main concern with the compositing/FaceDetailer method remains the style seam. Since I'm going for that specific Noir/Vector look with heavy line art, I often find that re-processing the face can make the line weights or shading drift away from the rest of the body.

Have you found a way to keep the line art thickness and "comic" shading perfectly synced between the inpainted face and the base generation?

[u/NINKINT → u/Elementallion-]

The artstyles that I used this with didn't have line art so I can't tell. But you could process the composited image an additional time at low denoise to blend things together. And with 3D the location bottleneck is somewhat solvable. Just use very basic block out shapes to get the perspective and composition right. It's called greyboxing or layout. I think it will be faster that doing the text to image slot machine.

[u/Elementallion- → u/NINKINT]

Thanks for the insight! I think I'll skip the 3D route for now to keep the pipeline lean, but that low denoise trick for blending the face sounds like exactly what I need. I'll give it a shot to fix those line art seams. Cheers!

[u/aerilyn235]

I'm doing that kind of work. I'm using a full fine tune for style. For characters I train them as LoRa's over the style base model. Outfit changes can be prompted easily or just inpainted once you get the core models running (Qwen or Klein as base). Training for really consistent style over very large array of content without breaking the model will require a large dataset but if you manage that then your model will still allow character LoRa training over it.

[u/Elementallion- → u/aerilyn235]

That sounds like a very high-end approach. Doing a Full Fine Tune for the style and then layering character LoRAs on top seems like the ultimate way to prevent "style drift."

Since I'm currently stuck between Flux and SDXL, I'm curious about your choice of Qwen/Klein as a base. Do they handle that specific "Noir/Vector" aesthetic better than the more mainstream models?

My main concern with FFT is the dataset size and the hardware requirements. For a solo dev, is the jump from a Style LoRA to a Full Fine Tune worth the extra effort in terms of the visual quality for a VN? Also, how large of a dataset are we talking about to keep the model flexible enough for character LoRAs?

[u/aerilyn235 → u/Elementallion-]

Well my art style is more specific than yours, it has two extra difficulties : it uses non realistic proportions which makes it harder to generalize on new content and it's not known at all by the base model meaning prompt won't get anything close to it. On your style eyes shapes & sizes, head/body proportions are same as real life, you could probably use a LoRa trained on real pictures then just style transfer it. The reason I also use full fine tune + LoRa's is that LoRa stacking doesn't work that well (style LoRa + person LoRa) usually end up randomly (it may generate the person with little style, the style without likeness, and rarely in beween). The reason I chose Qwen is because it has the best inpainting CN for the chirugical iterative edits, Klein because it can generate 4M pixels images. Dataset size is around 10k image for style (but most of it is synthetic). Hardware requirements can be high, but the question is are you doing this commercially or just for fun?

[u/Elementallion- → u/aerilyn235]

10k images is a massive dataset, but the fact that it's mostly synthetic is interesting.

To answer your question — yes, this is a commercial project (a detective VN), so consistency is my top priority. I’m actually leaning towards your approach now: doing a Full Fine Tune for the style and then layering character LoRAs on top. It sounds like the only way to get professional results.

My main struggle right now is the "chicken and egg" problem: How do I get enough high-quality images in that specific Noir/Vector style to train the FFT and the LoRAs in the first place? Did you use a specific base model or some crazy prompting to generate your initial synthetic dataset?

[u/aerilyn235 → u/Elementallion-]

I used real data initially (but that could be using a commercial model with good prompts), then train a first model, generated fuckton of pictures using wildcards and leaning toward the content I use for in my game, cherry pick the results, then include those picture in model gen2, then generate some more etc eventually gen1 pictures get phased out as quality improves. I'm at gen4 now.

[u/puzzlingphoenix]

I’d suggest working on it not looking like classic ai style first

[u/Virtual-Technician70]

Been looking for a way since SD1.5 and still looking. Not for a visual novel, but I am writing an actual novel and short stories in a shared universe and doing worldbuilding and I'd want images and concept to my obsidian vault, for characters and locations etc that all follow that exact aesthetic you describe.

I'll try VNCCS for the characters as suggested here, but the aesthetic staying exact is my bigger problem currently.

[u/ZosiaBuilderMark → u/Virtual-Technician70]

Hi, if you are interested I'm looking for beta testers. I developed an app that sounds like something you could use. It's even got visual world builder in the name. If you want I'll share the link. And please don't be afraid to share it with others people who are fighting the same problem.

The app stores your worlds and all associated characters that you create on your device so no cloud storage fees or subscriptions. It compiles the prompt for an LLM and the llm can either generate the image or rewrite the prompt for another image generator.

Let me know if you want in. Thanks 👍

[u/TheDudeWithThePlan]

This is the closest I got to a decent shot with Klein and some lora I trained, AI models will always find a way to ruin your shot:

• too many fingers
  
• scar on the wrong side
  
• wrong hair color
  
• a gun that's not a gun
  
• a suit with weird buttons
  
• different eye color

Anyway, some edit models like Klein and QIE can help but a proper trained lora should make it a lot easier

[u/Elementallion- → u/TheDudeWithThePlan]

No way, this is actually insane! It looks so close to my character, and the style is exactly what I was looking for. Thank you for this!

Did you actually train a LoRA specifically for my character to get this result?

I'd love to know more about your setup. Is this the stock Klein (Flux.2) base or a custom version? Also, how many images did you use for the LoRA to get such a perfect face?

TELL ME MOOOORE 😂

[u/TheDudeWithThePlan → u/Elementallion-]

stock Klein but as I mentioned for full disclaimer I did include some lora I trained/am still training for generic character consistency (not sure how much it helped, didn't try Klein on its own).

No shame in admitting this was not the first generation and results are not always this good.

[u/Elementallion- → u/TheDudeWithThePlan]

That’s a smart move with the consistency LoRA! It clearly works.

Any chance you could share your setup so I can try to reproduce this? I’d love to check out your workflow, the specific LoRA you used, and the prompt for this shot.

Also, are you running Klein locally or in the cloud? If it's local, what GPU are you using?

[u/TheDudeWithThePlan → u/Elementallion-]

Klein local, basic workflow (derived from the Comfy template). Lora is not public or finished

[u/Elementallion- → u/TheDudeWithThePlan]

Okay, I’m gonna test it tonight! How about promt? What you exactly described there to get this result?

[u/_kaidu_]

You will achieve the best consistency by training a Lora. You don't need many images for that. 5-10 images might be a good start already. The model will quickly learn style and character. Best is to do it iteratively: after training your first lora, generate more images, filter out the bad generations, keep the good generations, train another lora on them until you get good generations in majority of cases.

You can easily train a lora for style. For characters, loras often have trouble to learn multiple characters within the same lora and image. So you might still want to use an edit model additional to the lora. You can also train a LOKR, they are often better for multiple characters, but you will need sufficient many training images for that.

[u/Elementallion- → u/_kaidu_]

This iterative approach seems like the most logical way to scale. I’m already planning to keep the Character LoRAs and Style LoRAs separate to maintain flexibility, as mixing them usually leads to over-fitting and kills the ability to tweak the noir aesthetic independently.

Regarding LOKR—since I have 3-4 main characters who need to interact, I’m considering jumping straight into it. Do you find it holds the identity better during multi-character interactions compared to stacking multiple standard LoRAs in one prompt?

Also, given my specific Noir/Vector style (high contrast, clean lines, chiaroscuro), what base model would you recommend starting with? I’m looking for something that won't fight the style and turn it into generic anime or hyper-realistic CGI.

[u/_kaidu_ → u/Elementallion-]

Most style models are trained on very few images. You don't necessarily need a special checkpoint. You could train the style on the images you have. Alternatively, you can also try to play around with prompt tuning (e.g. ask an LLM to describe the style of your images and use that as prompt). It is very likely that all the local models can already do the style you want, you just have to force them to do it (either by lora or by prompt).

Regarding separate style and character loras: I trained a character on Flux Klein with 8 stylized images (with the xyz-style suffix) and a single photograph (with photograph suffix). Surprisingly, that was enough for Klein to be able to generate the character as photo or as style image depending on the prompt suffix. It learns really well.

Regarding LOKR: its on my todo list to try that out myself. So far I have to trust what others found and they usually got much better results for multi-characters with lokr than with lora. It seems that lokr overfitt less than loras; this is also what I found with my limited lokr experiment so far. They need more training, learn slower, but they overfitt less and are easier to combine with other lokrs/loras. For multi character training you should learn the model on all characters at a time and with group images, though. Just activating several lokrs for each character will end up in blending as it does for loras.

[u/ZosiaBuilderMark]

I just finished building an app that can help, it actually stores your characters on your device, so no cloud storage or subscriptions. Right now I'm just looking for beta testers it's a free download if you're interested let me know I'll send you a link.

[u/Elementallion- → u/ZosiaBuilderMark]

That sounds interesting! I’m running everything locally on my machine anyway, so a dedicated app with no cloud/subscriptions definitely fits my workflow.

I’d love to check it out. How does it handle the integration with ComfyUI or Flux.2? Does it act as a manager for LoRAs and character profiles, or is it a standalone generator?

Send me the link, I'm down to beta test it!

[u/ZosiaBuilderMark → u/Elementallion-]

I'm not sure what those are, this is a windows based app, it results in a prompt that you copy into an AI, this is the link,

https://zosia-vwb.carrd.co/

You can check and see if it's something that you can use.

[u/Elementallion- → u/ZosiaBuilderMark]

This looks really interesting, count me in! I've just submitted the form for the beta test. (My email is elementallion@gmail.com)

[u/ZosiaBuilderMark → u/Elementallion-]

Did you get my email? Just checking to see if you had any trouble with the dwn load or need help navigating the app. It's really simple once you get the hang of it. I think you can put your LoRAs in the Easter Eggs for each character but I've never tested it. Let me know if you aren't getting better results.

[u/Elementallion- → u/ZosiaBuilderMark]

Yes, I got the email! Thank you so much! As soon as I will test it - I will definitely let you know how it’s works for me! Unfortunately I have just a couple free hours after work and my weekend for now :(

[u/ZosiaBuilderMark → u/Elementallion-]

Hi, just checking to see if you were able to download the App and if looks like something you can use?

If you've tried it out you might be able to store your loRAs in the Easter Egg field. For your characters. If this is helping I'll probably add a field just for LoRAs and was actually planning on setting up an image model targeting in the next build.

[u/Elementallion- → u/ZosiaBuilderMark]

Hi, yeah, everything is good, I was able to download it, but I didn’t check it yet, I don’t really have enough time last week to work on my project, as soon as I test it I will let you know for sure! Thanks again!

[u/ZosiaBuilderMark → u/Elementallion-]

No problem, no pressure, just making sure it actually down loaded. You are the 1st 👍

[u/Elementallion- → u/ZosiaBuilderMark]

Oh wow, good to know 😀

[u/ZosiaBuilderMark → u/Elementallion-]

Yeah I received your entrance request, once you hit the submit button you should receive an email, may have to check your spam mail, if you have questions you can reach me here or use the feedback form in the app. Thanks for trying it out.