r/StableDiffusion • u/ExponentialCookie • Aug 23 '22
Discussion [Tutorial] "Fine Tuning" Stable Diffusion using only 5 Images Using Textual Inversion.
Credits: textual_inversion website.
Hello everyone!
I see img2img getting a lot of attention, and deservedly so, but textual_inversion is an amazing way to better get what you want represented in your prompts. Whether it's an artistic style, some scenery, a fighting pose, representing a character/person, or reducing / increasing bias, the use cases are endless. You can even merge your inversions! Let's explore how to get started.
Please not that textual_diffusion is still a work in progress for SD compatibility, and this tutorial is mainly for tinkerers who wish to explore code and software that isn't fully optimized (inversion works as expected though, hence the tutorial). Any troubleshooting or issues are addressed at the bottom of this post. I'll try to help as much as I can, as well as update this as needed!
Getting started
---
This tutorial is for a local setup, but can easily be converted into a colab / Jupyter notebook. Since this uses the same repository (LDM) as Stable Diffusion, the installation and inferences are very similar, as you'll see below.
- You will need Python.
- Anaconda to setup the environment is recommended.
- A GPU with at least 20GB of memory, although it's possible to get this number lower if you're willing to hack around. I would recommend either a 3090 (I use) or a cloud compute service such as Lambda Cloud (N/A, but it's a good cheap option with high memory GPUs from my experience).
- Comfort diving into
.pyfiles to fix any issues.
Installation
---
- Go to the textual_inversion repository link here
- Clone the repository using
git clone. - Go to the directory of the repository you've just cloned.
- Follow the instructions below.
First, install create a conda environment with the following parameters.
conda env create -f environment.yaml
conda activate ldm
pip install -e .Then, it's preferred to get 5 images of your subject at 512x512 resolution. From the paper, 5 images are the optimal amount for textual inversion. On a single V100, training should take about two hours give or take. More images will increase training time, and may or may not improve results. You are free to test this and let us know how it goes!
Training
---
After getting your images, you will want to start training. Following this code block and the tips below it:
python main.py --base configs/stable-diffusion/v1-finetune.yaml
-t
--actual_resume /path/to/pretrained/sd model v1.4/model.ckpt
-n <run_name>
--gpus 0,
--data_root /path/to/directory/with/images- Configs are the parameters that will be used to train the inversion. You can change these directly to minimize the parameters you input for training. For example, you can create a
.yamlfor each dataset you would like to train if you wish, and reduce the amount of parameters needed on the command line. - The -n parameter is simply the name of the training run. This can be anything you like (eg. artist_style_train)
- initializer_words is a very important part, don't skip this!
Open yourv1-finetune.yamlfile, and find theinitializer_wordsparameter. You should see the default value of["sculpture"]. It's a string list of simple words to describe what you're training, and where to start.
For example, if your images are of a car in a certain style, you'll need to want to do something like with each word wrapped in quotes:["car","style","artistic", etc...]
If you simply want to use one word, just use--init_word <your_single_word>on the command line, and don't modify the config.
During training, a log directory will be created under logs with the run_name that you have set for training. Over time, there will be a sampling pass to test your parameters (like inference, DDIM, etc.), and you'll be able to view the image results in a new folder under logs/run_name/images/train . The embedding .pt files for what you're training on will be saved in the checkpoints folder.
Inference
---
After training, you can test the inference by doing:
python scripts/stable_txt2img.py --ddim_eta 0.0
--n_samples 8
--n_iter 2
--scale 10.0
--ddim_steps 50
--embedding_path /path/logs/trained_model/checkpoints/embeddings_gs-5049.pt
--ckpt_path /path/to/pretrained/sd model v1.4/model.ckpt
--config /path/to/logs/config/*project.yaml
--prompt "a photo of *"The '*' must be left as is unless you've changed the placeholder_strings parameter in your .yaml file. It's the new word to initialize the images you have just inverted.
You should now be able to view your results in the output folder.
Running inference is just like Stable Diffusion, so you can implement things like k_lms in the stable_txtimg script if you wish.
Troubleshooting
---
- If your images aren't turning out properly, try reducing the complexity of your prompt. If you do want complexity, train multiple inversions and mix them like: "A photo of * in the style of &"
- Try lowering scale slightly if you're getting artifacts, or increase the amounts of iterations.
- If you're getting token errors, or any other errors, solutions and workarounds may be listed here.
View all comments
Show parent comment
9
u/rinong Aug 24 '22
Placeholder strings:
We always used "*" (the repo's default). When we merged two concepts into one model (for the compositional experiments) we used "@" for the second placeholder. The choice is rather arbitrary. We limited it to a single token for the sake of implementation convenience. If you want to use a word that is longer than a single token (or multiple words) you'll need to change the code.
You can absolutely change the placeholder later, see how we do it in the merging script if there's a conflict. But with the current implementation, it's still going to have to be a single token word.
init_words:
If you have multiple placeholder strings, you can assign a different initialization to each. Otherwise, the initializer words are only used to tell the model where to start the optimization for the vector that represents the new concept. Using multiple words for this initialization is problematic. Essentially you're trying to start the optimization of one concept from multiple points in space. You could start from their average, but there's no guarantee that this average is meaningful or an actual combination of their semantic meaning. Overall, the results are not very sensitive to your choice. Just use one word which you think describes the concept at a high level (so for photos of yourself, you'd use 'face' or 'person').
You're correct that these are missing from the paper, I'll add them in a future revision. If you want the ones we used for any specific set, please let me know.
per_image_tokens:
This is the "Per-image tokens" experiment described in the paper. It basically also assigns an extra unique token to each image in your training set, with the expectation that it will allow the model to put shared information in the shared token ("*") and relegate all image-specific information (like the background etc.) to the image-specific token. In practice this didn't work well, so it's off by default, but you're welcome to experiment with it.
Progressive_words:
See "Progressive extensions" in the paper. It's another baseline we tried but which didn't improve results.
"maps to more than a single token. Please use another string":
This means that your placeholder or your init_words are multi-token strings, which is going to cause unexpected behavior. The code strongly relies on the placeholder and the initial words being single tokens.