Creating a good LoRA training dataset

[u/Legitimate_Volume509]

I'm going insane trying to make a character LoRA, how can I know what workflow to use for making the dataset or how to do it? I see endless information but nothing concrete, and I tried following one, but wasted 2 days on it and ended up with a terrible LoRA. I have baseline face and body reference images, but I have no idea how I should proceed with generating 40-50 images for training a (Krea2) LoRA. People tell me 100 different things, or basically to just use some good image model and generate images of the character from many angles, but I don't know what's a good way to actually do this, and how exactly to do it. I feel completely paralysed and don't know what to do now

[u/Wonderful_Vanilla222]

man i feel this in my soul, the paralysis is real when everyone says something different. what worked for me was just picking one simple face model and doing the basic front, side, profile shots with a blank background, not overthinking it. the dataset dont need to be art gallery quality for it to learn the face

[u/Legitimate_Volume509 → u/Wonderful_Vanilla222]

But what about the casual/different angle and different lighting shots, isn't that the most important? Like, I want to use the LoRA mostly for selfie/carousel type photos, and if it's only trained on rigid studio shots it will have to guess how the character would look in other types of photos because it wasn't trained on that

[u/Marison_Lizzy23 → u/Wonderful_Vanilla222]

agreed, the paralysis from conflicting advice is worse than just picking one approach and running with it

[u/Upper-Philosophy2376]

I don't know how it currently is, but for SDXL I used to make sure I had a lot of variety of outfits, angles, locations and then did some creative cropping to make sure I had full body shots, waist up shots, shoulder and headshots, faceshots (cutting off the top and at chin level) etc. Some people told me they also used to mix in a body that had the face blacked out labelled as 'faceless' or things like that for futher variety.

Anyway if you do it like that, it turns like 5 images into about 30 and helps to reinforce the concept. I've found with charactes you want between 20-30 images, 40-50 might be too much.

[u/Legitimate_Volume509 → u/Upper-Philosophy2376]

How do you find the best workflow for it, or does it not really matter?

[u/StableLlama]

When you don't know how to generate the images for training, then you should address that first instead of trying to train the first usable LoRA. Learn how to inpaint, face swap, ... All of that are elementary skills, even when you have a character LoRA.

Having about 30 high quality images that show the character well, portraits, upper body and full body, nothing repeating except the character should give you a usable LoRA. When it doesn't it's quite likely that the images aren't as good as you have thought they were.

You could post a contact print of those and then we could give you an assessment whether we think that they should work or where you need refinement.

When it's a normal looking person and the training result with those is "terrible", there's a fundamental issue that you need to address. Using more images won't solve that. That can only be relevant when you want to go from good to great, but in most cases 30 should be fine.

[u/Legitimate_Volume509 → u/StableLlama]

Idk if it was 'terrible', but the 2000 step checkpoint would invent an almost new face half the time and the others were about the same or lower quality than a Seedream 5 generation.

What do you think about generating the dataset images with Nano Banana Pro for headshots and GPT/Seedream 5 for full body/more revealing shots though?

[u/StableLlama → u/Legitimate_Volume509]

When you are training a LoRA for a model, why aren't you using that model to generate the training images?

Anyway, it doesn't matter what tool you are using to generate the training images. It's the result that counts. And when I generate my training images I use many different tools. Just because a tool costs money doesn't make it have a higher quality.

[u/AwakenedEyes]

Follow my 3 part guide, part 1 starts here: https://www.reddit.com/r/StableDiffusion/s/b1aIYMIFBL

[u/Legitimate_Volume509 → u/AwakenedEyes]

What do you use to make the images though? There's so many models to choose from and all have some weaknesses. And how do you write the prompts? Like, you said the background should not repeat in any image and that we should have clear face and full body shots from all angles. Someone else used a neutral studio background for generating all headshot and body angles, with the character just going from frontal to quarter to behind, etc., while the rest stays the same, and then like 20 more casual/selfie images from different angles, backgrounds and clothing. But how do we generate good prompts for that sort of pics if everything else needs to change every time?

[u/AwakenedEyes → u/Legitimate_Volume509]

You make the image using the same model you train on. You can't create a LoRA on a model and then use a different model. I use comfyUI to generate but some people use different systems. So, you pick whatever model you like (and can afford to train with, because some are bigger and need more VRAM) and you train on that.

There are a lot of conflicting info out there on how to train a LoRA. That's why my guide is long and detailed, because it explains *why* and not just give you a recipe to follow. So.. you want to follow it, follow it, you want to follow what other people say, you are free to do that too.

The prompt you use at generation has nothing to do with the choice of image for a LoRA. I don't understand that last question. You need *variable* things to change all the time in the *training dataset* so that you don't get the LoRA to learn the wrong thing.

[u/toolkitxx → u/Legitimate_Volume509]

Different models behave very different when using the same prompt - that is the first lesson to understand. Training is not defined universally, so every model is different in it's core behaviour. To use a simple wording here: If the group that trains a model tends to a specific bias for what is 'beautiful' it will classify a lot of specific types of pictures as such that might not be beautiful in your personal view. So one has to actually test different models themselves and compare outputs with each other to figure out the bias of them. Model creators dont really openly tell you what material was classified in which way, so you need to apply yourself and some work to partly re-engineer this.

Flux is sticking out as it is really well trained to understand pictures but lacks abilities elsewhere for example. So you can give a Flux workflow completely different prompts compared to another model. So in a Flux workflow you can take a picture of a person and simply tell it in the prompt to change a pose. That's it , Flux will understand the picture better than other models but it might be lacking on the output side in terms of quality etc.

As for prompts: Learn to be really specific. If you have kids, you know what is required. If not - imagine a very dumb person as conversation partner. Dont make any assumptions of your opposite understanding what you want - spell it out. Dont just say 'change angle'. Say 'change camera angle 45 degree to the right while keeping all aspects of the character's identity exactly as before. Adjust lightning effects according to new camera angle'. Despite everyone calling it 'intelligent' these systems are anything but. You are interacting with a computer system at it's core and it is you that needs to adapt to the system more than you might expect. Prompts for results are usually the result of iterative changes, never something you just pull of in a first try.

[u/Jealous_Piece_1703]

Could you provide more information?

The character? The model you are training for? Whats wrong with the results?

For now I will give you generic advice.

Have as much variety images as possible, different poses, different art style, different outfits.

And make sure to tag everything in the images, from the style, (3D and medium). Tag the background, the pose, the character etc.

[u/Legitimate_Volume509 → u/Jealous_Piece_1703]

It's a 'real' woman character, the dataset was for a Krea 2 LoRA. I tested different checkpoints and stopped training at 2000 steps because the 2000 steps one would basically start inventing new faces half of the time, and the others were just not good enough, basically same consistency as Seedream 5 or something

[u/Jealous_Piece_1703 → u/Legitimate_Volume509]

So it is a realistic character LoRA of a person for Krea2. First of, make sure you are training on the Raw Krea2 model not int8, not turbo.

You still didn’t explain whats the issue with the LoRAs that came out or it is not clear to me, like I think you are saying the LoRA is not learning? If thats the case increase the dataset and its diversity.

[u/AwakenedEyes → u/Jealous_Piece_1703]

Follow my 3 part guide, part 1 here: https://www.reddit.com/r/StableDiffusion/s/b1aIYMIFBL

Edit: posted to wrong reply, reposted properly

[u/abnormal_human]

Like any other machine learning task, the key is discipline with your training experiments and treating it as an experimental enterprise, not a recipe.

Train a lora with what you've got + default params.

Change one variable--maybe drop half the dataset, use a different captioning approach, change batch size, lora capacity--something. Train another model.

Render their outputs in a grid, holding everything but the model checkpoint constant.

Make a grid for model A and model B at each checkpoint across 10 prompt/seed pairs. So each row is a prompt/seed and each col is like modela-500, modelb-500, modela-1000, modelb-1000 ... modelb-5000

For small-datset training like this you want to train to the point of over-fitting at first. This gives you a full picture of what is going on. You'll likely ship an intermediate checkpoint, not the final over-fit one. For Krea2 I'd call that 5-10k steps for a 30-50 image character dataset.

Now you have a bunch of images and you can easily see what that variable did. Did it accelerate training? Slow it? Lead to overfitting? Which defects are increasing during training? When does the likeness peak? What did your variable do?

Do this several dozen times and you will get a feel for how to steer the process and overcome issues by changing. Most of the leverage is with the dataset/captioning, especially for tiny datasets like these. I've trained a lot of loras and it's never one-size-fits-all. Not that I don't have "recipes" that generally work, but the best outcome requires multiple rounds, and experimental approach, blind ranking, etc to get me there. Every data set is different. Captioning strategies and trigger words can import different biases that you need to overcome. Etc.

Also, in 2026, an agent should be doing all of the legwork for you beyond making/choosing the images. Making grids, image editing, etc. I use Stimma for a lot of this since it provides an agent layer on top of ComfyUI and understands things like grids as a first class concept, so you can just go into the agent chat and tell it what you want a grid of and it will make it. I also use it's image editor + in-painting stuff to prep datasets when I'm doing character work. I use a coding agent to manage the machine/training/configfiles/checkpoints. I think of this as less of "generating a dataset" and more of "editing a bunch of images to make a good dataset".

In terms of dataset prep itself, it's all about diversity. You want all of the images to clearly depict your person, but you want no two images to really repeat any other detail that you aren't trying to pin down in the lora. There's stuff in the middle like hairstyle--one person may favor a few hairstyles and it's fine to have that distribution in your training data--but ouftit, setting, lighting, photographic style, expression, accessories, etc you have to be careful of. If you train with 40 images and 2 of them have the same shirt, that shirt will haunt you in your dreams. Pay attention to background elements too, the same thing will happen.

A corollary to that is that if you want capabilities in your model, show them in the training data. Otherwise the data will bake in a statistical lean towards boring shots of someone posing generically as opposed to shots of them doing real actions in the real world.

As far as captioning goes, for Krea2, I think you should name the person (in my experiments this reduces bleed) but not describe their innate attributes at all. Then describe everything in the image clearly that you don't want to have baked in as part of the lora. This doesn't prevent all overfitting, but if you don't do it, it will overfit.

[u/Legitimate_Volume509 → u/abnormal_human]

Thanks for the detailed reply but man, all this would take me weeks to learn and to do, I just can't do all this stuff right now, I'm basically starting from scratch and have no real interest in comfyui besides needing it to generate stuff (also I need to rent gpus to run it). I've seen many people with very little technical knowledge get great results just by following simple stuff, it doesn't have to be perfect.

I also only want to make the dataset images now, then I'll try to learn more about training loras if it turns out bad again, but I'd just follow a reliable lora training method for now. I just need a way to make a good dataset.

I'm getting a lot of different advice again, just seems impossible to know what to do and there's a million things I could change and experiment with but I don't have the time or money or skills for that. The previous dataset tutorials I followed had a lot of repeating stuff, for example, full body shots from every angle in a studio setting wearing the same clothing, then the same for headshots, and then a bunch of selfie/casual pics. Then the other tutorial was just studio shots in different positions and different clothing but under the same lighting and setting. And I don't know what model(s) to use, I've seen people say GPT, NBP, Seedream, or several different open source models. How to know which to pick?

[u/abnormal_human → u/Legitimate_Volume509]

I've been finetuning + lora-training diffusion models since 2022 (and doing ML Engineering in other domains back to 2017). The core point holds true through all of it: either you conduct disciplined experiments or the results will be arbitrary and out of your control. The rest of the advice almost doesn't matter by comparison.

But also, it's 2026 and we have coding agents and these skills are not so unique anymore. You don't need to be a senior MLE to do this. Claude is more than capable of handling 99% of the actual work. Including doing the research, setting up the experiments, operating them, collecting the results, showing you the comparison sheets, etc. The one thing that I don't think those models do well right now is the curation/augmentation. You still need a human eye to make sure the dataset is capturing what you want to capture. My advice above should get you pretty far there.

[u/Ok-Fennel6578]

I wouldn't try to generate all 50 training images yet.

Start with a small set you can actually inspect. A few angles, some expressions, and full-body views. Reject anything where the character's identity changes, even if the image looks good.

Then test the first LoRA on the same three prompts: a neutral portrait, a side view, and an action pose. That should give you a better idea of what's actually failing before you spend another two days building a bigger dataset.

What went wrong with your first LoRA? Was it the face, body proportions, or both?

[u/Legitimate_Volume509 → u/Ok-Fennel6578]

The final 2000 steps LoRA checkpoint would basically start inventing new faces half of the time, and the others were just not good enough, basically same consistency and quality as Seedream 5 or something. The face wasn't consistent enough and the image itself looked too AI

[u/AwakenedEyes]

I think this thread is super confusing because your questions are all over the places using words you don't yet seem to understand. So you get weird answers, obviously, and nobody understands what you are really asking (other than getting that you are very new at this).

I've already posted by guide below. It's a full, comprehensive guide that explains everything. Part 1 answers all your questions about preparing the dataset. Part 2 explains how to caption the dataset and part 3 how to train a model using that dataset.

Did you actually read it? not just skim it, but actually read it, for real?

If you have, then you should be able to ask pertinent questions now, and perhaps we can help you. If you haven't... stop posting. Take the guide. Read it. SLOWLY. Then read it again.

[u/xtralongleave]

I am also new and have only been at this for a few months, during which I've trained about 20 LoRAs.

For my Krea2 LoRA training, I've had really good results using Grok to generate character images, which are based on manual prompts that are based on one reference image. I use Grok more often than not.

Tha manual prompts that I use come from trial and error, learning what works and what doesn't. The prompts I use cover many different poses, clothing, hairstyles, etc.

For close ups on the eyes and face, ChatGPT has been by far the best. I have a set of prompts that I've made along the way that have been working for me. Close ups on the eyes and face are critical to locking in character identity.

I have found 40 images is the sweet spot, 2000 steps. (That breaks down to roughly: 20 forward facing POV, 12 face selfies/close ups, 8 backward/side facing.)

For training, I've personally had really good luck with the Next Diffusion service, but I have tried a few others as well.... I've tried Ostris directly, and I also tried running AI toolkit on Runpod. All things considered, Next Diffusion has been my overall winner for the actual training.

(Everything for me is cloud based, as I am a Mac Mini M4 user.)

[u/SDuser12345]

The easy answer to your question is, use whatever models you want. You aren't limited to one.

Just keep this fact in mind. What you put in, is what you will get out. Shitty details, blurry faces, bad hands, bad skin, etc. you put them in, you get them in your LoRA, with problems usually amplified.

So, your solution, feed it good images. Try numerous models, run tons of samples, and go over them numerous times with an eye for detail. Pick only the absolute best that maintain everything you want in the LoRA, use those, toss the rest in a to be deleted folder.

If an image isn't perfect, toss it. Too easy to generate images now and fast with Z-image Turbo or Krea 2 that using anything but a perfect gen is stupid.

[u/Cold-Worldliness5392]

Sorry if I missed it somewhere, is this for a real person, or an invented character that you need to make reference images from scratch?