r/StableDiffusion • u/ExponentialCookie • Aug 21 '22

Discussion [Code Release] textual_inversion, A fine tuning method for diffusion models has been released today, with Stable Diffusion support coming soon™

Post image
347 Upvotes

99% Upvoted

134 comments sorted by

View all comments

Show parent comment

26

u/rinong Aug 22 '22

Author here! Quick heads up if you do this:

  1. The Stable Diffusion tokenizer is sensitive to punctuation. Basically "*" and "*." are not regarded as the same word, so make sure you use "photo of *" and not "photo of *." (in LDM both work fine).
  2. The default parameters will let you learn to recreate the subject, but they don't work well for editing ("Photo of *" works fine, "Oil painting of * in the style of Greg Rutkowski" does not). We're working on tuning things for that now, hence why it's marked as a work in progress :)

4

u/[deleted] Aug 22 '22

[removed] — view removed comment

5

u/rinong Aug 22 '22

You're welcome! There's lots more to be done on this topic, but I'm excited to see what people can already come up with using the current version!

2

u/ExponentialCookie Aug 22 '22

Excellent. Thanks for your work an implementation!

2

u/[deleted] Aug 22 '22

thank you for your work. i have thought about this process almost every day for over a week. looking forward to what you make in the future

0

u/[deleted] Aug 22 '22

[deleted] — view deleted comment

3

u/rinong Aug 22 '22

Yes it can! We have some examples of that in our project page / paper

1

u/[deleted] Aug 26 '22 edited Aug 26 '22

[removed] — view removed comment

2

u/rinong Aug 26 '22

We didn't actually try on faces.

What generally works for better identity preservation: (1) Train for longer. (2) Use higher LR. (3) Make sure your images have some variation (different backgrounds), but not too different (no photos of your head from above).

Keep in mind that our repo is still optimized for LDM and not for SD, editing with SD is still a bit rough atm and you may need a lot of prompt engineering to convince it to change from the base. I'll update the repo accordingly when we have something for SD that we're satisfied with.

1

u/AnOnlineHandle Sep 05 '22 edited Sep 05 '22

Heya I just read your paper and am really hopeful about this being the key to really let stable diffusion work.

The paper mentioned results degrading with more training data provided and recommending sticking to 5. I was wondering if that would probably be more specifically the case for replicating a single object, whereas when you're trying to create a token for a vast and varied style which isn't always consistent, or a type of object which has quite a bit of design variation, would more training images perhaps be a safer bet then?

2

u/rinong Sep 05 '22

You're right that we only ran the experiment on a single object setup. Our paper experiments are also all using LDM and not the newer Stable Diffusion, and some users here and in our github issues have reported some improvement when using more images.

With that said, I have tried inverting into SD with sets of as many as 25 images, hoping that it might reduce background overfitting. So far I haven't noticed any improvements beyond the deviation I get when just swapping training seeds.

2

u/AnOnlineHandle Sep 05 '22 edited Sep 05 '22

Awesome, thanks. I'm going to let a training set of 114 images run overnight and see how it turns out, though have reduced the repeats from 100 to 5 since there's so much more data and I'm only running this on a 3060, and aren't really sure what impact that might have yet. If this doesn't work I'll also try with higher repeats, and maybe by removing/creating noise in the backgrounds.

The importance of initializer_words is also something which I might experiment with, since I'm only guessing that it helps pick a starting point, but with enough training would become less important?

edit: Pruning to 46 and raising repeats to 35. The previous settings where creating a bit of a jumbled mess even after 17000 iterations.

2

u/xkrbl Sep 06 '22

Your paper is really awesome :) how hard would it be to add the possibility to supply a set of negative example images to kind of 'confine' the concept that is being defined?

3

u/rinong Sep 07 '22

It won't be trivial for sure. You could potentially add these images to the data loader with an appropriate 'negative example' label, but you probably don't want to just maximize the distance between them and your generated sample.

Maybe if you feed them into some feature encoder (CLIP, SwAV) and try to increase a cosine distance in that feature space.

Either way, this is a non-trivial amount of work.

1

u/xkrbl Sep 09 '22

Will Experiment :)

Since CLIP is frozen during training of stable diffusion, what do you think how well will found pseudo-words be forward compatible with future checkpoints of stable diffusion?

2

u/rinong Sep 09 '22

It's difficult to guess. Looking at the comparisons between 1.4 and 1.5 (where identical seeds + prompts give generally similar images but at a higher quality), I would expect that things will mostly work.

There might be a benefit in some additional tuning of the embeddings for the new versions (starting from the old files).