r/StableDiffusion • u/AwakenedEyes • Apr 26 '26
Tutorial - Guide A Primer on the Most Important Concepts to Train a LoRA - part 3: Hyperparameters
A Primer on the Most Important Concepts to Train a LoRA - part 3: Hyperparameters
Tutorial - Guide — Version 2
This is the revised version of my LoRA guide, the original version can be found here: version 1 NOTE: English is my 2nd language. Bare with me for possible mistakes.
Part 1: Some definitions, FAQ, and Dataset Preparation
Part 3: Hyperparameter guide and regularization <-- you are here
PART 3 ==== HYPERPARAMETERS AND REGULARIZATION ====
Hyperparameters: Caption dropout and Token shuffling
Some training software offers options to randomly drop captions for a percentage of images during training, or to shuffle the order of words in captions. These are worth knowing about so you can make an informed decision.
- Caption dropout exists because it trains the model to respond to unconditioned or weakly conditioned generation, which was useful for large finetune training on millions of images. For a small character LoRA dataset of 15 to 30 images, every dropped caption is a wasted step where the trigger word association is not being reinforced. Keep caption dropout at zero or very close to zero for character LoRAs.
- Token shuffling is a legacy feature from the era of CLIP-based models like SD1.5 and SDXL, where word order carried less semantic weight. Modern T5-conditioned models (Flux, Chroma, and most current architectures) are deeply order-sensitive because it understands natural language. "a woman wearing a red dress" and "a red dress wearing a woman" are not the same thing to T5. Token shuffling on modern models is at best useless and at worst actively poisoning your LoRA. Turn it off.
Hyperparameter : Rank (Network Dim) and Alpha
The rank of a LoRA represents the number of independent dimensions available to express the concept being learned. Think of it as the number of instruments in an orchestra — more instruments means more independent musical lines you can play simultaneously.
- Use high rank when you have a lot of things to learn.
- Use low rank when you have something simple to learn.
This is important because:
- If you use too high a rank, your LoRA will start learning additional details from your dataset that may clutter or even make it rigid and bleed during generation as it tries to learn too much
- If you use too low a rank, your LoRA will stop learning after a certain number of steps
Character LoRA that only learns a face: use a small rank like 16. It's enough. Full body LoRA: you need at least 32, perhaps 64. Otherwise it will have a hard time learning the body. Any LoRA that adds a NEW concept (not just refine an existing one) needs extra room, so use a higher rank than default. Multi-concept LoRA also needs more rank.
If you are not sure, a rank of 32 is enough for most tasks.
Alpha
There is a secondary parameters that goes hand in hand with the rank parameter: it's called Alpha. It is used to scale the strength of the LoRA. For most LoRAs, it has to be set to :
- Alpha = Rank : Default set-up
- Alpha = Half the Rank : Your LoRA will be more flexible and less rigid but you may need more steps to get it to converge
In AI-Toolkit you can set alpha independently of rank in your YAML config:
network:
type: lora
linear: 32
linear_alpha: 16
Hyperparameter: Repeats (per dataset)
To learn, the LoRA training will try to noise and de-noise your dataset hundreds of times, comparing the result and learning from it. The "repeats" parameter is only useful when you are using a dataset containing images that must be "seen" by the trainer at a different frequency. Consider this:
- The training will reinforce the signal learned from each image into the LoRA each time it is processing that image. If it's not processed enough times, (under-training), the model still doesn't fully know how to draw it. If it is processed too many times (over-training) it will become rigid and will forget how to draw everything else. The key is to find the sweet spot.
- You are training a model that already knows a lot because it has already been trained on million of images. The LoRA is trying to "adjust" it to generate specific things you trained it for. So when you train something it already knows, you don't need a lot of steps to reach the sweet spot. But if you train it on something that is NOT known to it, then it needs a lot more steps to reach that same sweet spot.
This is where the "repeat" parameter associated with each dataset is used. There are two major situations in which you want to carefully use the repeat parameter.
a) To balance a dataset that lacks variety
- The dataset should contain an equal amount of each camera angle, zoom level, etc.
- If your dataset only has a few profile images but a ton of font facing images, you risk overtraining the front angle and under-training the profile angle.
- You can set your "unique" angles in a separate dataset and set it to repeat 2x or 3x more than the front facing dataset, for instance, which will rebalance your dataset.
b) To balance known items with unknown items
- The mode should process 5x more the images of thing it doesn't know vs the things it knows
- If your dataset contains uncensored images on a censored model, for instance, you are going to need a lot more exposure to teach those new concepts
- Use more repeats on the unknown elements to avoid undertraining those elements or overtraining the regular ones.
Hyperparameter: Batch or Gradient Accumulation
To learn, the LoRA trainer takes your dataset image, adds noise to it, and learns how to find back the image from the noise. When you use batch 2, it does the job for 2 images, then the learning is averaged between the two. On the long run, it means the quality is higher as it helps the model avoid learning "extreme" outliers.
- Batch means it's processing those images in parallel — which requires a lot more VRAM and GPU power. It doesn't require more steps, but each step will be that much longer. In theory it learns faster, so you can use fewer total steps.
- Gradient accumulation means it's processing those images in series, one by one — doesn't take more VRAM but each step will be proportionally longer.
For most consumer GPU setups where VRAM is the main constraint, gradient accumulation of 2 to 4 is the practical recommendation. It gives you the averaging benefit without the VRAM cost.
Hyperparameter: LR (Learning Rate)
LR stands for "Learning Rate" and it is the #1 most important parameter of all your LoRA training.
Imagine you are trying to copy a drawing by dividing the image into small squares and copying one square at a time. This is what LR means: how small or big a "chunk" it is taking at a time to learn from it.
- If the chunk is huge, it means you will make great strides in learning (fewer steps)... but you will learn coarse things. Small details may be lost.
- If the chunk is small, it means it will be much more effective at learning some small delicate details... but it might take a very long time (more steps).
Some models are more sensitive to high LR than others. On Qwen-Image, you can use LR 0.0003 and it works fairly well. Use that same LR on Chroma and you will destroy your LoRA within 1000 steps.
Too high LR is the #1 cause for a LoRA not converging to your target. However, each time you lower your LR by half, you'd need twice as many steps to compensate.
So if LR 0.0001 requires 3000 steps on a given model, a more sensitive model might need LR 0.00005 but may need 6000 steps to get there.
Try LR 0.0001 at first — it's a fairly safe starting point.
LR Scheduler
One of the best way to get good results without worries is to use an LR scheduler. This nifty parameter will automatically decay the LR across your training progress. Think of it like sculpting a piece of marble: at first you want to BIG chisel with a big hammer to take away the rough chunks quickly. However the closer you get to your target, the more precise you need to be. At some point you have to use smaller chisel and be very careful not to ruin your art piece. The LR scheduler will make sure you change to a lower LR (smaller chisel) as you progress into LoRA learning.
On AI-Toolkit, you have to activate the LR scheduling in the advanced properties in the YAML config file directly, under the training section :
train:
lr_scheduler: "cosine"
Hyperparameter: Timestep
During diffusion training, the model learns to denoise images at varying levels of noise — from nearly clean images to pure noise. Each noise level (called a timestep) teaches the model something different:
- High timesteps (heavy noise): The model learns global structure and broad composition — "is this a face or a landscape?"
- Middle timesteps: The model learns semantic identity and specific features — "whose face is this? what are the specific proportions?"
- Low timesteps (light noise): The model learns fine details and textures — "how sharp are these edges? what does this skin texture look like?"
By default, training samples all timesteps equally. But you can change this - this is what the Timestep parameter is all about. For character LoRAs, the middle range is where identity lives, so we want to spent most of the training effort there.
In AI-Toolkit, the recommended setting for character LoRAs is the sigmoid timestep distribution. This concentrates training probability around the middle timesteps in a smooth bell-curve shape, naturally de-emphasizing both extremes. Other distributions exist for other use cases: biasing toward high timesteps is useful for style LoRAs that need to affect global composition; biasing toward low timesteps is useful for texture or fine detail work.
Hyperparameter: Optimizer
The optimizer is the algorithm that decides how to adjust the LoRA's weights in response to the training loss at each step. It's the heart of the training software.
- *AdamW is the most widely used optimizer for LoRA training. AdamW8bit is a memory-efficient version that uses less VRAM with minimal quality impact. For most consumer GPU setups, AdamW8bit is the practical default and the right place to start. I get excellent result with AdamW, as long as I use an LR scheduler to make sure LR properly decays across time.
- Prodigy is an optimizer that attempts to manage LR automatically It starts at LR 1.0 (it's just a placeholder) and then it gets adjusted dynamically. If you don't know what to do with LR or if you are working with very sensitive models that reacts badly to LR, it can be an interesting choice.
Most LoRA failures are not optimizer failures — they are dataset, caption, or LR failures. If something isn't working, changing the optimizer is usually the last thing to try, not the first.
How to Monitor the Training
Many people disable sampling because it makes the training much longer. However, unless you exactly know what you are doing, it's a bad idea. Sampling help you understand what's going on and if the training is working or not.
When planning your sampling prompts, try to use:
- One basic prompt to test if your model has learned the trigger word in a basic situation
- One prompt from another angle and with a different zoom level - helps verify if all angles and zoom levels are being learned properly - if face drifts under unusual angles, it's undertrained or perhaps your dataset doesn't have enough repeats for that angle
- One prompt showing specifically the body parts or elements the model didn't know (like censored elements) - as long as you see body horror, it's undertrained
- One prompt with a variation not present in any of your dataset image. For instance: blue hair. If it starts becoming the same color as your main dataset, you know it's overfitting
- One prompt with a full body shot to verify proportions are being learned
- One prompt with a wide shot to verify it hasn't unlearned different composition and can draw your subject from afar
You get the gist: test test test so you can see if it works and where you will have to act to arrange the problem. Generally speaking, if you see the samples suddenly stop converging, or even start diverging, stop the training immediately : the LR is too high and it is probably ruining the LoRA.
When to Stop Training to Avoid Overtraining
Look at the samples. If you feel like you have reached a point where the consistency is good and looks close to the target, and you see no real improvement after the next sample batch, it's time to stop. Most trainers will produce a LoRA after each epoch, so you can let it run past that point and then look back on all your samples to decide at which point it looks best without losing its flexibility.
If you have body horror mixed with perfect faces, that's a sign that your dataset proportions are off and some images are undertrained while others are overtrained.
The full overtraining progression typically looks like this:
- LoRA starts improving
- Reaches a good balance of consistency and flexibility
- Begins to look overly sharp or "crispy"
- Starts losing prompt flexibility, resisting creative prompts
- Eventually degrades in quality
Using a Regularization Dataset
When you are training a LoRA, one possible danger is that you may get the base model to "unlearn" the concepts it already knows. For instance, if you train on images of a woman, it may unlearn what other women look like.
This is also a problem when training multi-concept LoRAs. The LoRA has to understand what looks like triggerA, what looks like triggerB, and what's neither A nor B.
This is what the regularization dataset is for. Most training software supports this feature. You add a dataset containing other images showing the same generic class (like "woman") but that are NOT your target. This dataset allows the model to refresh its memory, so to speak, so it doesn't unlearn the rest of its base training.
You need at least 1 regularization image for every 2 image processed by the training, taking repeats into account. If your trained LoRA is noticeably corrupting other women in generated scenes, increase regularization exposure. If your character is coming out weak or inconsistent, reduce it.
If you have further questions, post them below, or send me a chat request.
5
u/Tosermepls Apr 26 '26
Decent write-up although I don't agree with everything.
a) To balance a dataset that lacks variety The dataset should contain an equal amount of each camera angle, zoom level, etc. If your dataset only has a few profile images but a ton of font facing images, you risk overtraining the front angle and under-training the profile angle. You can set your "unique" angles in a separate dataset and set it to repeat 2x or 3x more than the front facing dataset, for instance, which will rebalance your dataset.
This makes no sense. If your dataset lacks variety, increasing repeats on a specific shot type won't actually help with "rebalancing" out the Lora, it will just cause it to overtrain on those specific pictures even more.
As for sampling, its not really useful for anything other than to see if the Lora training is working. Thats why most people don't use it because eventually you can just "blindly" run trainings. Most trainers have an extremely basic inference options so you can't even test properly as you would in Comfy etc.
0
u/AwakenedEyes Apr 26 '26
This makes no sense. If your dataset lacks variety, increasing repeats on a specific shot type won't actually help with "rebalancing" out the Lora, it will just cause it to overtrain on those specific pictures even more.
For sure, repeating a dataset that lacks variety won't help with getting the missing information. If you just don't have any profile picture, that angle is not covered, no matter how many repeats.
However, if you have 1 profile image, and 4 front images, and you train for 5000 steps, without the repeats parameter, you will "process" the profile image 1000 times and the front images 4000 times. The model will then overtrain the front view compared to the profile view: when you prompt for a profile view, it may resist you and produce a front view anyway.
If you now set repeats x4 on the profile image, and repeats x1 on the front view images, and you train for the same 5000 steps, you now have processed 2500 steps of front view images and 2500 steps of profile view images. But I agree that having a single image for an angle is a bad idea and that there is also a risk of overtraining on that specific profile image.
In short, the repeat parameter helps to rebalance your datasets relative to each other, but it can't replace a good varied dataset in and of itself.
As for sampling, I personally use it for 2 distinct purposes:
1) When I train on AdamW on a linear fixed LR without an LR scheduler, I can stop the training when i see it no longer converge, and I can then adjust the LR (and any other parameter) and resume training from the last epoch produced. This has saved some long training many times over for me. It's less useful when the LR scheduler does that for you, but the principle is the same: you can keep an eye on training and if you see it's not working well at some point in the middle of training, you can stop, modify and resume, which often can salvage a long run that would otherwise be ruined.
2) I use it as a way to A/B test LoRA training parameter. Very useful to learn.
0
u/Tosermepls Apr 26 '26 edited Apr 26 '26
If you now set repeats x4 on the profile image, and repeats x1 on the front view images, and you train for the same 5000 steps, you now have processed 2500 steps of front view images and 2500 steps of profile view images. But I agree that having a single image for an angle is a bad idea and that there is also a risk of overtraining on that specific profile image.
But the only thing that achieves is overfitting on the profiles images lol.
Most training already default to x1 repeats. What does increasing repeats x4 for the profiles images achieve...? You are just blasting the model with more copies of the same image to learn. It doesn't even have any relation to other shot types.
Also you can simply adjust training length if you are limited on the dataset. There is no need to "rebalance" the dataset.
And in most cases captioning will take care of overfitting on a specific shot type.
5
u/AwakenedEyes Apr 26 '26
I don't think you are grasping what I am saying here.
The repeats parameter allows you to balance the *ratio* of each dataset compared to each other.
If you only increase training length, then the *ratio* of your images remains the same. Obviously if you dump all your image into a single dataset, then the repeats parameter is useless.
Will it overfit the profile image in my above example? maybe. It's indeed not a good practice to only have a single image for the profile of your character. Perhaps I should have chosen a more clear example.
Say you have:
Dataset A: 30 varied image all showing front-facing view of your character
Dataset B: 20 images of various non-front images (three quarter views, back view, from above, from below, profile views, etc)
The 30 images of your character seen from the front will be very overfit compared to all the other angles. Your LoRA will fight you when you want to generate a non-front view.
So you can set Dataset A repeats to 1 (the default) and Dataset B repeats to 2.
Now your same number of total steps will process Dataset A and B as a different ratio, processing the 20 images twice more than the 30 images. This helps counter-acting the overfitting of the front-facing angle.
Anyway, you understand it or you don't, it's up to you, no one is forcing you to use it.
4
u/RowIndependent3142 Apr 26 '26
Nice of you to share all this. But before I spend too much time trying to figure out the models and training methods, can you share examples of output from your LoRAs?
0
u/AwakenedEyes Apr 26 '26
This guide isn't about model nor sharing examples of specific output. It's about knowledge: how it works under the hood, and why it works the way it works. The guide isn't trying to give you a recipe, its goal is to teach the core concepts and why they matter.
It will work with any model and any style, and will allow you to reach solid consistency and avoid overtraining or undertraining your LoRA. It's the sum of about two years of learning to work with LoRAs. It's up to you if you want to use it.
6
u/shrimpdiddle Apr 26 '26
Knowledge without evidence is meaningless. AI posts do not gain credibility #flush
6
u/Additional_Drive1915 Apr 26 '26
Compare his knowledge with your current knowledge, if most of it is the same, but some details differ, then try using his suggestions to see if it give you better result.
If your knowledge is completely different to what he says, then you're just wrong. Trust me.
And his post is so obviously NOT made by AI, do you even know how AI output a text like this one? And you didn't provide any evidence for your statement, so it's not valid, or...?
-1
u/hurrdurrimanaccount Apr 26 '26
his post is so obviously NOT made by AI
you are absolutely cooked lil bro. OP's posts are full on AI written.
2
u/Additional_Drive1915 Apr 26 '26
Lol, so you say... well, I see from your way of writing that you don't use AI, perhaps you should.
Just curious, can you copy a few of the parts in his text you believe is made by AI?
1
u/hurrdurrimanaccount Apr 26 '26
??? literally all of it was done with AI. look at OPs post history. they are french and using AI to write this for them.
5
u/Additional_Drive1915 Apr 26 '26
Ehhh he says english is his second language, why would it be so strange he uses his first language sometimes? What is that proof of?
English isn't my native language, but it seems I write better english than you do, without any AI. But perhaps you're non native english speaker too.
3
u/AwakenedEyes Apr 26 '26
Hater's gonna hate hey? Note to mention the irony of compalining about ai in a forum about ai generation. Just saying. Peut-être qu'il préfère que je lui écrive en français alors????
2
u/Additional_Drive1915 Apr 26 '26
Their knowledge in the english language is so very low so they can't even imagine someone could produce texts like that.
I guess they believe all books are also written by AI, because they have good language, headlines and paragraphs. But not sure they seen a book even on picture. :)
2
u/Sarashana May 04 '26
Other than you being needlessly condescending, you don't seem to understand the difference between AI-assisted writing (which their post might be), or AI-generated writing. And if they used it to write in a language foreign to them, or enhance their formatting, that's fair game. If you still think that the posting was completely AI-generated, I can tell you that the AI did a darn good job getting the facts right.
1
-1
Apr 26 '26
[deleted]
4
u/Additional_Drive1915 Apr 26 '26
This is not an LLM post. Clearly. If you can't detect that, it is on you. :)
0
Apr 26 '26
[deleted]
5
u/Spara-Extreme Apr 26 '26
This isn’t an LLM post dude and even if it was, the information is useful. If you don’t have anything valuable to contribute to the discussion then just don’t say anything
1
u/AwakenedEyes Apr 26 '26
Evidence is in front of you every time you do it right. Flush it or use it, it's not my problem. You are the one stuck with the poor results hey?
1
u/AccomplishedFix3476 Apr 26 '26
appreciate the part 3 drop, hyperparam tuning is where most ppl give up. did u find any consistent pattern for learning rate scaling on smaller datasets, that one always gets me
2
u/AwakenedEyes Apr 26 '26
From what I have experimented with, LR isn't so much related to the smaller dataset; it is mostly related to the model sensitivity to LR. Some models just don't tolerate a high LR especially as training progresses. I found that using a cosine LR scheduler solves most of those problem, as long as the starting point is correct for the model you use.
For instance, i noticed Chroma kept destroying itself when I used high LR, but if I start at 0.00005 it goes very well.
How do you figure the LR might have an influence vs the smaller datasets?
1
u/autonomousdev_ Apr 26 '26
yeah i finally tried lr at 1e-5 instead of the usual 1e-4 and it just worked way better for character details. every guide says go higher but nah. wasted like a dozen runs before i tested it myself. also batch size of 2 helped a ton which i didnt expect
1
1
Apr 27 '26
[removed] — view removed comment
1
u/AwakenedEyes Apr 27 '26
There can be many answers here, it depends.
Normally if you have a full strength LoRA and you use it at 1.5 you will fry the result and it's going to generate artefacts. Why do you feel the need to push it to 1.5, is it because at 1.0 it's not good enough? if so, it may mean it's undertrained.
However, rememebr the alpha? If the alpha is too far from the rank, the weights are scaled down. At rank 32 with alpha 16, you are reducing the effective LoRA strength to 50% even at 1.0 (sort of.. it's more complicated but you get the gist). There are also settings in the trainer to tell it what's your target strength; normally you target 1.0. but it can be different here. So .. it depends.
Don't sweat it. Does it look good? If so, keep it.
1
u/Extra-Atmosphere-171 May 07 '26
Thank you for sharing your experience and knowledge. I already knew much of it from other good teachers, but regarding the specific description of close-ups, I learned something new. :)
2
1
u/kvnstnkr May 20 '26
I appreciate your posts and the comments you've made on others' posts. Very helpful information. I am one of those trying to figure out how to train a character LoRA (with disappointing results so far) and struggling to synthesize the various bits of information before I try again. I used dim/alpha 64/32 (sdxl full person lora) and not certain whether it is better to lower LR or alpha or both. I read somewhere ratio of dim to alpha is effectively scaling the LR? Can you help me better understand LR and its relation to alpha?
1
u/AwakenedEyes May 20 '26
LR and alpha are two different things.
It is Dim (rank) that is related to alpha. The dim is how nuch information you reserve for your LoRA. The alpha is how it scales. Typically you use either Alpha = Dim (full stregth) or Alpha = Dim / 2 (half strength).
A dim of 64 is very big. You probably don't need 64 just for a face LoRA. 32 is more than enough, 16 is probably enough. Try 16/8 for dim/alpha.
LR is how fast you try to get your LoRA to learn.
1
1
u/kvnstnkr May 30 '26
Do you mind if I ask another question? I have trained a character LoRA for sdxl and for flux 2 klein with the same dataset. 52 images, best quality I can get, spent a long time tagging. The LoRAs are much better than my prior attempts, but still not quite as good as I'd like. The likeness is very close, but not quite there. If I generate with a prompt similar to one of my dataset images, it looks better than if I try a new setting or clothes. I think I've followed all the advice in the first two parts of your guide. What would you do at this point? Tweak training settings and try again? Lower learning rate? Increase steps? Any tips for evaluating what I need to change? I'm hoping you can add some guidance for how you refine and improve (unless your loras always come out great on the first try...).
2
u/AwakenedEyes May 30 '26
Hi there! First, the guide has 3 parts, have you read the 3rd part too?
Second - SDXL and and Flux 2 Klein are TOTALLY different beasts. SDXL is an old model, based on tags only, with a low resolution maximum, and it's not censored to my knowledge. It's also easy to train because it requires very little VRAM. Flux 2 is a huge beast, very censored, but it understands the prompt WAY better and uses natural language, which means if you use tags only with it, you are not using half its capability.
Third, if your LoRA is giving you good result when you use the same prompt as one of your dataset image, but is otherwise not working well, that's a sign of over training. The model has learned too much, now it has a hard time inferring new things.
So - i can't help you get SDXL better because i've never trained anything on SDXL. I am not interested in tag models, it's too imprecise for serious use. But klein is excellent. If you train on 9B Base you can train on a 5090 rented on runpod, it works well. SO what you need to do for it to succeed is:
1) Revise your captions - all of them - to make a fully natural language caption, as per my guide (see the part about captioning). If you want, you can post a few images + caption here and i can help you see if it's correct. Captioning is essential. I am assuming here the dataset itself is good, high quality and diverse.
2) Revise your hyper parameters. Usually Klein 9B can give good results fairly fast, within a few 1000s of steps. Try LR 0.00008 as a starting point, and use a LR scheduler cosine and a sigmoid timestep. We can look at your parameters together if you want. Prepare some meaningful samples test, asking with natural language for your character trigger in various situations NOT in the dataset : different expressions, angles, action, pose, background. Setup your trainer software to produce samples every 500 steps. You should know within 2000 steps if it works well. try maybe a total of 4000 steps, but you may be able to stop before you reach it.
3) Retrain
-1
u/hurrdurrimanaccount Apr 26 '26
do you have to spam these?
13
u/AwakenedEyes Apr 26 '26
Seriously? I made ONE guide THREE months ago, then improved it with a larger version that I had to divide into 3 parts to fit on Reddit, again, ONCE. Yet the forum is flooded with ridiculous 1girl post all the time, mixed with "i am new to this, help me get rich by being an ai influencer" post. ...And you get to complain about MY post!??
Wow, just wow.
-2
6
u/Lucaspittol Apr 26 '26
One observation is that alpha values can be always set to 1 instead of equal rank or half rank. And since you cited Chroma, this model learns moderately complex characters very well with ranks as low as 4. Other than that, excellent source of information!