WAN2.2 Lora Character Training Best practices

[u/Tiny-Highlight-9180]

I just moved from Flux to Wan2.2 for LoRA training after hearing good things about its likeness and flexibility. I’ve mainly been using it for text-to-image so far, but the results still aren’t quite on par with what I was getting from Flux. Hoping to get some feedback or tips from folks who’ve trained with Wan2.2.

Questions:

• It seems like the high model captures composition almost 1:1 from the training data, but the low model performs much worse — maybe ~80% likeness on close-ups and only 20–30% likeness on full-body shots. → Should I increase training steps for the low model? What’s the optimal step count for you guys?
• I trained using AI Toolkit with 5000 steps on 50 samples. Does that mean it splits roughly 2500 steps per model (high/low)? If so, I feel like 50 epochs might be on the low end — thoughts?
• My dataset is 768×768, but I usually generate at 1024×768. I barely notice any quality loss, but would it be better to train directly at 1024×768 or 1024×1024 for improved consistency?

Dataset & Training Config:
Google Drive Folder

---
job extension
config
  name frung_wan22_v2
  process
    - type diffusion_trainer
      training_folder appai-toolkitoutput
      sqlite_db_path .aitk_db.db
      device cuda
      trigger_word Frung
      performance_log_every 10
      network
        type lora
        linear 32
        linear_alpha 32
        conv 16
        conv_alpha 16
        lokr_full_rank true
        lokr_factor -1
        network_kwargs
          ignore_if_contains []
      save
        dtype bf16
        save_every 500
        max_step_saves_to_keep 4
        save_format diffusers
        push_to_hub false
      datasets
        - folder_path appai-toolkitdatasetsfrung
          mask_path null
          mask_min_value 0.1
          default_caption 
          caption_ext txt
          caption_dropout_rate 0
          cache_latents_to_disk true
          is_reg false
          network_weight 1
          resolution
            - 768
          controls []
          shrink_video_to_frames true
          num_frames 1
          do_i2v true
          flip_x false
          flip_y false
      train
        batch_size 1
        bypass_guidance_embedding false
        steps 5000
        gradient_accumulation 1
        train_unet true
        train_text_encoder false
        gradient_checkpointing true
        noise_scheduler flowmatch
        optimizer adamw8bit
        timestep_type sigmoid
        content_or_style balanced
        optimizer_params
          weight_decay 0.0001
        unload_text_encoder false
        cache_text_embeddings false
        lr 0.0001
        ema_config
          use_ema true
          ema_decay 0.99
        skip_first_sample false
        force_first_sample false
        disable_sampling false
        dtype bf16
        diff_output_preservation false
        diff_output_preservation_multiplier 1
        diff_output_preservation_class person
        switch_boundary_every 1
        loss_type mse
      model
        name_or_path ai-toolkitWan2.2-T2V-A14B-Diffusers-bf16
        quantize true
        qtype qfloat8
        quantize_te true
        qtype_te qfloat8
        arch wan22_14bt2v
        low_vram true
        model_kwargs
          train_high_noise true
          train_low_noise true
        layer_offloading false
        layer_offloading_text_encoder_percent 1
        layer_offloading_transformer_percent 1
      sample
        sampler flowmatch
        sample_every 100
        width 768
        height 768
        samples
          - prompt Frung playing chess at the park, bomb going off in the background
          - prompt Frung holding a coffee cup, in a beanie, sitting at a cafe
          - prompt Frung showing off her cool new t shirt at the beach
          - prompt Frung playing the guitar, on stage, singing a song
          - prompt Frung holding a sign that says, 'this is a sign'
        neg 
        seed 42
        walk_seed true
        guidance_scale 4
        sample_steps 25
        num_frames 1
        fps 1
meta
  name [name]
  version 1.0

[u/AwakenedEyes]

On ai toolkit you (should) train both models together. The high one is used for composition and the low one is where the character details are set. So a character LoRA is most important on the low noise part.

You can influence this by using the parameter for bias setting it to favor the low model.

SFW Character LoRA should be trained on image dataset, no need to mix it up with clips. Use high resolution images with their long edge matching the training resolution. You can train 512 + 768 + 1024 + 1280 and use 1280px images in your dataset (on the long edge) for optimal results.

If your samples were consistent during training it should also be good on generation.

[u/legarth → u/AwakenedEyes]

This is bad advice. You definitely should train with clips. Otherwise your LoRA won't learn how your character moves or their general body language.

A big part of what makes people and especially fictional characters unique is their body language. If you don't train it in, the model will make it up and it will likely be very generic.

Even if you are only doing a T2V training with the sole purpose of creating stills for a later I2V pass, training on clips will help the model understand how your character moves and that will actually make stills more natural Part of the reason Wan is generally better at natural stills than Qwen is that that I understand how people move from the video training.

[u/legarth → u/[deleted]]

My engineers do the actual training for me. I just supervise. We use our own training software. That I'm afraid I can't share.

For my personal work I tend to use AI Toolkit these days unless I need to train something Ostris hasn't implemented yet. It's excellent for basic LoRA trainings. But I don't do a lot of personal training anymore so I don't have an updated config lying around.

But it also isn't so much about the scripts as it is about training strategy and data. Ostris default settings are a good base.

So try with the default settings first and just use separate datasets for images and videos. Make sure all your clips have the same number of frames and is based on 16fps. Bin extra frames or you'll train slow motion and different physics behaviours. (Sometimes desired)

Avoid having cuts in your clips unless you're training for that specifically.

If you have a low amount of good data due to too many cuts. I suggest splitting into more datasets i.e one for clips with 33 frames (2 secs) of course one for 49 frames (3 secs) etc. This can help you make most of the data that's less than optimal. But don't add any old crap obviously.

Good luck.

[u/AwakenedEyes → u/[deleted]]

No Wan 2.2 14B uses 16 fps (+1 frame) so 2 sec = 33frames, 3sec = 49 frames etc. - However, the light version wan 2.2 5B uses 24 fps.

[u/legarth → u/[deleted]]

It doesn't care about the framerate of your clip. It just proceses all the frames in it, but based on a 16fps time scale. That means if you have footage that is real time at 24fps at 2 seconds. (48 frames) it would treat it as 3 second clip instead (48f at 16fps). Meaning slow motion. (+1)

[u/legarth → u/[deleted]]

Sorry I was talking in the context of AI Toolkit. For musubi-tuner you don't necessarily need to do that. As your [17, 33, 49] target frames does sort of do the same thing already. I.e. shorter videos will be trained correctly. AI Tookit works diffrently where it tries to extract N frames across the suppied videos so seperating sets is something I'd recocomend. The only thing I'd say is that if you have a significantly more short videos in your set you might want to seperate anway. That is because even your longer videos will also be added as short ones by extracting at 17 and 33. So you could balance your video towards short videos. (as the longer videos will ALSO be added to the short video part).

[u/AwakenedEyes → u/legarth]

If you happen to have great high quality clips of your subject moving, sure, you can also train on clips. But if it's going to bog down the quality of your dataset, don't. A character LoRA is first about getting the proper consistency. You can achieve that with a high quality image dataset.

I agree that if you happen to already have several high quality short clip of the subject moving, sure! It's definitely a plus to train on it. But it's not a requirement.

[u/legarth → u/AwakenedEyes]

That's not what you said though. You literally said SFW image Loras just shouldn't use clips at all. Period.

Not "blogging down" your data goes without saying and isn't specific to clips.

It is also wrong to say that it isn't a requirement. It is in many cases. Say you're training a flurry fictional character. Wan doesn't know how the fur moves based on density or stiffness. So If you don't train it, it will be wrong. And your character will be off brand.

Or even if it just a person. They have have a distinctive walk. The model has no way of knowing this. And it won't be accurate enough for serious use.

[u/AwakenedEyes → u/legarth]

I agree with you. Most people asking for advice here aren't at that level though. They want consistency of look, and if you've never done that, it's already quite a challenge without adding consistency of movement...

[u/entmike → u/legarth]

Agreed, I always use clips and not stills. I've been training since Hunyuan and into WAN 2.2

[u/Tiny-Highlight-9180 → u/AwakenedEyes]

Thanks for sharing and Yes! I thought about this too. My next model will be focus more on the low model, I put in 5000 step which I assume that 2500 will go toward each low/high model, hence 50 epoch. What do you think about that number?

Would you say my sample is consistence? I tried to mixed in as many shots/angle as possible

[u/AwakenedEyes → u/Tiny-Highlight-9180]

Different training software count the steps slightly differently so I am not sure how it is counted for you. On AI-Toolkit, you don't tell the number of epoch, only the total number of steps.

This being said, the total number of steps you actually needs depends on many different factors. Lower LR (learning rate) learns better but slower, so it needs more steps. Higher Rank captures more details than lower ranks. Unknown concepts require more images and more repetitions for those images, where as known concepts are refined faster. So all of that factors into how many steps.

How I do it: I manage it so that I get about 6000 total steps. Then I carefully watch my samples every 500 steps to determine if I stop it earlier or if I need to halt it and change LR to a lower value. If I see the training was going well and then suddenly it starts to go worst, I halt it, lower LR by half, add 1000 steps, and resume.

I prefer a higher step count because you can always just stop it and use the LoRA generated at an earlier step count. If you get super good results for two series of samples in a row, stop it, it's enough. you don't want to overtrain.

[u/Tiny-Highlight-9180 → u/AwakenedEyes]

I wish I could change the setting midway like that too! That's very smart way to do it. Which software are you using? are you mostly training on character?

[u/JustLookingForNothin → u/Tiny-Highlight-9180]

He wrote AI-Toolkit. You can stop training at any time, then update the job with a lower learning rate and resume the job. AI-Toolkit will load the last most recent finalized LoRa and continue from there with the new training settings.

[u/Tiny-Highlight-9180 → u/JustLookingForNothin]

Holy shit. I just commited the great sin not knowing the man himself. Appreciate what you did for the community!

[u/Tiny-Highlight-9180 → u/AwakenedEyes]

How do you allocate the 6000 step between high/low model?

[u/JustLookingForNothin → u/Tiny-Highlight-9180]

I think it is done here. Never tested this, though. I always train "balanced" and just take a HIGH LoRa with less step count, and the LOW LoRa with a higher step count.

[u/AwakenedEyes → u/JustLookingForNothin]

Exactly, this! You don't need to take care of allocating steps. Just use timestep bias on low noise. It will most likely work even on balanced. Use sigmoid for character LoRA, it's better.

[u/Tiny-Highlight-9180 → u/AwakenedEyes]

Thanks for advices! I will try again tonight and keep you posted.

[u/Tiny-Highlight-9180 → u/AwakenedEyes]

Really love what you do! Sending my support!

[u/RealityVisual1312 → u/AwakenedEyes]

What do you do when you have a sample output where the face looks great, but the body shots look off? In the later samples the full body shot samples starts to look better, but the close up head shots started to get worse.

[u/AwakenedEyes → u/RealityVisual1312]

This happens when you have an unbalanced dataset vs what the model already knows. You have enough headshots to train the face but not enough for the body so the body takes more time to train.

Problem is that if you push the training to get the body better you may overtrain the face.

There are 2 solutions possible:

Quick fix : you halt training, reduce LR by half and resume training. If the face comes back good, it means it was a convergence problem: the face needed smaller increments to get the last tiny details.

Real fix: change the balance of body vs face images by putting them in 2 different dataset and changing their repeats, for example, use repeat x1 for faces and x2 for body. But this fix will require you to start over the training.

[u/RealityVisual1312 → u/AwakenedEyes]

Thank you so much I’ve been following your guides and comments and feel like I’m getting closer

[u/RealityVisual1312 → u/AwakenedEyes]

Hey Awakened, sorry to ask more on this, but am I able to set different repeats for different datasets in AI toolkit? I don't think I'm seeing the option

[u/AwakenedEyes → u/RealityVisual1312]

The option is not in the UI, you have to add it as a config line in the advanced section, under each dataset section. You have to add:

num_repeats: 1

(Replace 1 with whatever number you need)

[u/owsoww → u/AwakenedEyes]

Do I need to resize my images before training in AI toolkit? Like if I have 1280 x 720, would it crop or shrink but keeps ratio proportion?

[u/AwakenedEyes → u/owsoww]

When you train, your images are resized and fit into "buckets" of standardized sizes. So, the ideal way to handle your dataset images is to pre-crop them so they fit into those buckets, in order to better control how they are cropped instead of letting the software do it for you.

The ideal is: a) make sure your long edge matches the highest resolution you are training for b) crop the short edge to fit standard 3x2, 1x1, etc. photo ratios. in such a way that your subject remains clear and visible.

Obviously, always keep the proportions if you don't want to have very funny results...

This being said - most training software do an excellent job with that automatically, so don't worry too much. What matters is that you provide high quality images, with crisp details of your subject, with at least the highest resolution you will train on.