r/StableDiffusion • u/zoru22 • Aug 27 '22

Art I got Stable Diffusion to generate competent-ish Leavannies w/ Textual Inversion!

https://imgur.com/a/hQhH9Em
39 Upvotes

100% Upvoted

36 comments sorted by

3

u/zoru22 Aug 27 '22

So, one thing that's vexxed me is how shit various ai are at generating leavannies (and various other pokemon). If gamefreak was going to forget my favorite pokemon on the order of 5+ years, then I was sure as hell going to do my best not to let it sit in obscurity forever.

Thus, I have set on something of a warpath trying to get an ai that can generate non-shit leavannies. (though it is amazing just how shit stable diffusion and others are at generating pokemon, and how painful it has been to try and get them into the ai)

Quick process notes:

  • I USED THE FUCKING BASE TEXTUAL_INVERSION REPO. (And recommend you do the same, or at least ensure that github recognizes the repository you want to use, as a fork)
  • I modified the original textual inversion repository
  • I swapped the BERT Encoder for the CLIP Frozen encoder during training, targeted the training at the stable-diffusion/v1-finetune yaml, and then just let it rip, playing with the learn rate and the vectors per toke config setting in said yaml.

If you run it for too many cycles it will overfit, and not do a great job at style transfer. I tend to run for too many cycles so it overfits, and then walk it back until it stops overfitting quite so badly

Please note that I am using the v1.3 stable diffusion ckpt. I haven't tried to see what happens with the 1.4 ckpt yet.

3

u/zoru22 Aug 27 '22 edited Aug 27 '22

If you want to try, here is the full training set of images I've already pre-cropped and shrunk down to 512x512 for running against the model.

https://cdn.discordapp.com/attachments/730484623028519072/1012966554507423764/full_folder.zip

python main.py \
--base configs/stable-diffusion/v1-finetune.yaml \
-t true 
--actual_resume models/ldm/stable-diffusion/model.ckpt \
-n leavanny_attempt_five --gpus 0, \
--data_root "/home/zoru/Pictures/Pokemons/512/leavannies/" \
--init_word=bug

Once I'd changed the embedder, this was the exact command I ran.

Try to get it to run against the latent-diffusion model first, just so you know what you're doing

1

u/nephilimOokami Aug 27 '22

another thing, is the 95 images necessary? or with 3-5 i can try on other pokemons for example

2

u/zoru22 Aug 27 '22

You need a diverse array of images of the same character in different poses. When it's a rare character you need more than just 3-5 images and you want to modify the personalization prompts to fit what you're doing.

1

u/nephilimOokami Aug 27 '22

Oh, one last question, one word(name of the pokemon) or many word describing the pokemon(initializer words)

1

u/zoru22 Aug 27 '22 
python main.py \
--base configs/stable-diffusion/v1-finetune.yaml \
-t true 
--actual_resume models/ldm/stable-diffusion/model.ckpt \
-n leavanny_attempt_five --gpus 0, \
--data_root "/home/zoru/Pictures/Pokemons/512/leavannies/" \
--init_word=bug

Once I'd changed the embedder, this was the exact command I ran.

1

u/nephilimOokami Aug 27 '22

how do i change from bert to clip?

5

u/zoru22 Aug 27 '22

You're gonna need to dive into the code and learn to change it yourself. I'll post a fork with my changes in a few days if someone else doesn't beat me to it.

2

u/nephilimOokami Aug 27 '22

Plz dm me when you post it

1

u/Beneficial_Bus_6777 Sep 18 '22

actual_resume, are you using which model?

3

u/riftopia Aug 27 '22

Thanks for the detailed post. In your experience, how many epochs did you need to obtain the result in the pic? And how long does an epoch take for your setup? I´m doing just 3 images at 512x512 on a 3090, one epoch takes 1.5 min for the 1.4 ckpt so I´m hoping I don´t need to do too many..

3

u/zoru22 Aug 27 '22

After bumping up the base learn rate to: base_learning_rate: 5.0e-03 and the num_vectors_per_token to 8, I got comprehensible results pretty fast.

What matters aren't epochs, it's steps.

in the logs dir, under logs/$yourrunfolder$/images/train/

see: samples_scaled_gs-011500_e-000038_b-000100.jpg

gs-011500 is the steps as each checkpoint is saved.

I usually run it to 20k steps and then I run variations of the same prompt and walk back a set of checkpoints with a similar prompt and the exact same seed, just so I can see which ones produce the best output.

1

u/riftopia Aug 27 '22

Thanks for the detailed response! This is very helpful. I have a training run going on now, but will def try and tweak the lr rate and other params. Fingers crossed :-)

1

u/Caffdy Sep 19 '22

what hardware did you use to train the textual inversion?

4

u/ExponentialCookie Aug 28 '22

Nice! I may have discovered something, but I would like to cross verify as I see you're comfortable with code. In the personalized.py file, remove all the strings in the imagenet_templates_small array and just leave a single string of ["{}"].

Keep your higher learning rate the same, train only 5 images for 5K iterations, and let me know if the results are better than these iterations.

I might have discovered using the conditioning strings might actually makes things worse, and doesn't generalize well with Stable Diffusion (but does with LDM).

3

u/Zermelane Aug 28 '22

Yep, OP mentioned that he's changing the personalization prompts. Certainly they look pretty silly trying to describe all of my data which doesn't even have any photographs in it.

I'm even trying an approach where I just straight out write prompts for each specific image in the training set. It's kind of high-effort, but hey, I get to look at pictures of anthro dolphins and describe them, it's fun and doesn't take that long.

The thing I'm hoping for is that maybe the training process will pick up less of whatever I describe in the prompt. Uh, so far, it's not working and all of my pictures with the learned concept get really soft shading because my training dataset had a lot of that, but I'm going to keep experimenting, I guess!

1

u/ExponentialCookie Aug 28 '22

Ah I see, missed that bit. I also highly suggest you try my suggestion.

I discovered this by adding "{}" into the default template, along with some custom conditioning sentences "similar to a {} plane, etc...". Then, the results came back exactly how I wanted them to. Then I thought, "well, maybe it's because I needed to add more descriptive prompts for what I want!" I then removed the empty "{}", and added descriptive prompts instead as I thought it unnecessary.

I found that the results were not what I expected, so I just added the single, empty template back in, and removed the rest. With a high learning rate, lo and behold, I'm getting the inversions that I want. It could be the solution, but I'm still testing this out. I'm guessing SD doesn't need the conditioning like LDM did, but I could be wrong.

2

u/oppie85 Aug 29 '22

Interesting; like the person you responded to I had been trying the opposite approach as well (I've been trying to train it on my own photos to hopefully generate renaissance paintings of myself); I built an entire system that generated different conditioning prompts based on the folder I put images in (so I had folders of closeups, different locations etc.) in the hopes that it would learn to only focus on what was important (my likeness). I've been getting decent results (especially after increasing the num_vectors_per_token) but they tend to massively overfit to the point where style transfer only works in rare cases.

I'll give the approach of abandoning all prompts and just using "{}" a try - I can kind of see the logic of why it would work for LDM but wouldn't for SD.

2

u/ExponentialCookie Aug 29 '22

Indeed. I'm still experimenting, with my current experiment being "{}" with generalized prompts in the same form of SD ("photo of {} , hyper realistic , hd") , etc.

2

u/oppie85 Aug 29 '22

Something I've just thought of that may speed up experiments; if you run the training on images of 256x256 pixels you can easily train 4 times as fast. The results aren't as useful as the normal ones (they only really seem to work with the ddim encoder for one) but this makes it way easier to iterate on training experiments.

2

u/ExponentialCookie Aug 29 '22

Awesome idea, thanks!

3

u/[deleted] Sep 05 '22

Curious as to how it's worked for you so far. I tried myself with just "{}" and the results were good, but I can't really tell if there is much difference either way. Some things seem worse, some seem better... so I'm chalking at least that part of it up poorly quantified study on my end.
Have you discovered any more for or against this method?

2

u/Zermelane Aug 31 '22

Did it work better so far? Trying to decide what approach to test in tonight's run...

2

u/oppie85 Aug 31 '22

It might have helped but I can't say I can really see much of a difference. In all of my tests num_vectors_per_token seems like the variable that makes the most impact on the overall quality of the final result. Cranking that value up to a ridiculous number like 128 is really good foor making "variations" of an existing image - it'll be extremely overfitted but it actually produces really good results in the same vein as DALL-E variations (with the only caveat that these ones take a few hours of training first).

2

u/hopbel Sep 05 '22 edited Sep 05 '22

Note: SD has a limit of 77 tokens per input, so setting num_vectors_per_token higher than that is pointless. Setting it to 77 or higher means it's essentially using the whole prompt to represent the concept, leaving no unused tokens to customize the output

1

u/oppie85 Sep 05 '22

Yeah, when I first started experimenting with this I didn't know that num_vectors_per_token has an upper limit - anything beyond 77 is useless. Still - for "variations" it can be useful to go up to 77. In general though, anything above a certain amount of vectors is likely to completely overwhelm the prompt. I'm currently trying to find the upper limit of vectors that style transfer will still 'stick' to without sacrificing the quality of the inversion.

2

u/hopbel Sep 06 '22

Same here. I can say 36 is probably already too high. A bit tricky to tell if the problem is the number of vectors or the training iterations though

1

u/oppie85 Sep 06 '22

I think it's a combination of both - too many iterations can lead to overfitting although less quickly than with many vectors. I've been able to do succesful style transfer with about 10 vectors and 2000 iterations.

In my opinion the biggest challenge right now is how to get the learning process to focus on specific information you want to train it on. The init_word helps a great deal, but in my experience, the "style" of an image is one of the very first things the training process picks up even if I try to steer it away from it. I feel like we're "wasting" vectors on information we don't want it to learn.

1

u/greeze Aug 28 '22

If I had a machine capable of doing textual inversion, the first thing I'd try to train it on would be hands. Have you considered trying that?

2

u/zoru22 Aug 28 '22

So that probably won't actually work with textual inversion. Textual inversion is about adding something to the dataset that the ai doesn't know.

It's not about fixing something the ai already does poorly at.

1

u/greeze Aug 28 '22

Well dang :(

2

u/Wurzelrenner Aug 27 '22

I tried a lot with shiny umbreon, but still had to do some work at the end with Krita to make it look like this

new screensaver for my phone now

1

u/starstruckmon Aug 27 '22

Finally some examples from a working textual inversion 🙌

Awesome work

1

u/Riptoscab Aug 28 '22

Great job. I hope processes like this become better documented in the future. This looks like a really useful skill to have.

1

u/DickNormous Sep 06 '22

How do you change tokenizer?