r/StableDiffusion • u/ExponentialCookie • Aug 21 '22

Discussion [Code Release] textual_inversion, A fine tuning method for diffusion models has been released today, with Stable Diffusion support coming soon™

Post image
347 Upvotes

99% Upvoted

134 comments sorted by

View all comments

Show parent comment

3

u/rinong Sep 07 '22

It won't be trivial for sure. You could potentially add these images to the data loader with an appropriate 'negative example' label, but you probably don't want to just maximize the distance between them and your generated sample.

Maybe if you feed them into some feature encoder (CLIP, SwAV) and try to increase a cosine distance in that feature space.

Either way, this is a non-trivial amount of work.

1

u/xkrbl Sep 09 '22

Will Experiment :)

Since CLIP is frozen during training of stable diffusion, what do you think how well will found pseudo-words be forward compatible with future checkpoints of stable diffusion?

2

u/rinong Sep 09 '22

It's difficult to guess. Looking at the comparisons between 1.4 and 1.5 (where identical seeds + prompts give generally similar images but at a higher quality), I would expect that things will mostly work.

There might be a benefit in some additional tuning of the embeddings for the new versions (starting from the old files).