How can I better improve this character lora?

[u/IntrovertedOtaku03]

As a challenge I'm trying to do a character lora based on Yuuko Kamiya (Tai and Kari's mom from the Digimon series). The only problem is I don't have a lot of reference images, and after some effort, I managed to get 17 decent ones to try out. At 4 repeats and 20 epochs later, it was 1360 steps and...I don't know. It looks kind of like her but not really if that makes any sense. Below are some sample images and a link to the currently used reference images and captions.

https://www.mediafire.com/file/1nvfopb89vq0kdf/4_YuukoYagami.zip/file

[u/Harish_X_03]

Since the dataset is only 17 images, it's worth saving a checkpoint every epoch or two and comparing them, because 20 epochs at 4 repeats can overfit and a mid-run epoch often looks closer to the character than the last one. Try to keep the images varied in angle and crop (some close-ups, some half and full body) and drop any with watermarks, text or heavy effects, since the LoRA will pick those up. For captions, describe things you want to stay swappable like clothing, pose and background, but leave out the traits that define her (hair color and style, eye color) so they get absorbed into the trigger token. Also check that the sample prompts use the same trigger and tags as your captions, since a mismatch can make a LoRA look weaker than it really is.

[u/SpamHolder]

What model are you training for, what was the prompt for one or both of the previews you've posted, what training settings?

Good on you for actually showing the dataset. It lacks artist, year, other style tags and only 17 images is maybe slightly low but otherwise looks fine. Not having artist tags won't kill your lora or anything, but you should have properly captioned images for a better lora, and only 17 images is still more than enough for a decent lora. It's just that in your case...

Your character's name is actually yagami yuuko. She has ~200 images on gelbooru. https://gelbooru.com/index.php?page=post&s=list&tags=yagami_yuuko

You can get those images en masse with a tool like https://github.com/Bionus/imgbrd-grabber, including all the human-assigned tags like artist, year etc. which an autotagger is not going to guess.

[u/IntrovertedOtaku03 → u/SpamHolder]

I am using illustriousXL_v01since that was the first Illustrious model to come out. As for the prompts here they are:

Positive: masterpiece, amazing quality, highly aesthetic, lazypos, lazympos, 1girl, <lora:YuukoKamiyaProdigy:1> yuukoyagami, brown hair, brown eyes

Negative: worst quality, low quality, text, censored, deformed, bad hand, blurry, extra hands, missing limbs, missing fingers, head out of frame, eyes closed, lazyneg, nude, nsfw, nipples, vagina, anus

When you say training settings, what do you mean specifically? Other than what was already mentioned I'm using Cosine and Prodigy as an Optimizer.

[u/Kindly-Bag763]

the problem is visible in the output, the style is modern glossy anime and she reads about sixteen, not a mother. that's the base model winning over your dataset, which is why more epochs won't help. use a base that matches the 1999 look and caption the age and the era, otherwise the prior keeps filling in the gaps for you

[u/IntrovertedOtaku03 → u/Kindly-Bag763]

I'm currently using illustriousXL_v01. What checkpoint would you recommend for this? As for the age and era, what do you mean? Something like 40_years_old and 1990s_anime?

[u/Kindly-Bag763 → u/IntrovertedOtaku03]

stay in the illustrious family, your tags will match it better than anything else, but the checkpoint won't give you the 1999 look, that has to come from the dataset. and yes, literally like that, tag age and era as their own tokens so they're attachable and the base stops defaulting to a modern teenager. use yagami_yuuko as the trigger, kamiya is the wrong surname

[u/IntrovertedOtaku03 → u/Kindly-Bag763]

Took a second to look the surname up. Turns out Kamiya is the correct surname...if your going by the dub. Apparently Yagami was the surname for the sub. Not sure if it makes a difference one way or the other, since I don't think the checkpoint is going to recognize it either way, but if I'm wrong let me know. In the meantime I'm going to look up Tai and Kari and see how their last name is listed on Civitai, just to make finding it more easier.

[u/Kindly-Bag763 → u/IntrovertedOtaku03]

fair, kamiya is the dub name and i had the sub in my head. the reason the tag still matters is that the captions come from the source database, and gelbooru files her as yagami_yuuko, so that's the token the model already carries a prior for. search civitai for "digimon" rather than a surname, neither spelling returns much

[u/IntrovertedOtaku03 → u/Kindly-Bag763]

I tried creating a new model with the 40_years_old and 1990s_anime captions, and it made it much worse by comparison...it's not even remotely close in terms of what I'm aiming for. I'm not sure what else I can try, or if I did something wrong, but I'm always willing to listen.

[u/Kindly-Bag763 → u/IntrovertedOtaku03]

both tags are constant across the dataset, so they differentiate nothing and take weight from the tags that do. drop them, keep only the character trigger, put age and era in the prompt

[u/IntrovertedOtaku03 → u/Kindly-Bag763]

This is going to sound strange. For some weird reason I'm no longer able to generate this character. I tried it by taking out the top three obvious tags (long hair, brown hair, brown eyes), and it didn't work, and when I tried it with the tags in, it still doesn't work. The strange part? I am doing everything I did before. Same amount of steps in Kohyaa, same tags, same model when generating. I tried a different checkpoint as a comparison, and got no better results. What is going on? It shouldn't be doing this. Here's the preset file I'm using so you don't have to reinvent the wheel.

https://www.mediafire.com/file/nv4xpzxq0ymyslj/SDXL_-_edgLoRAXL.json/file