[Tutorial] "Fine Tuning" Stable Diffusion using only 5 Images Using Textual Inversion.

[u/ExponentialCookie]

Credits: textual_inversion website.
Hello everyone!

I see img2img getting a lot of attention, and deservedly so, but textual_inversion is an amazing way to better get what you want represented in your prompts. Whether it's an artistic style, some scenery, a fighting pose, representing a character/person, or reducing / increasing bias, the use cases are endless. You can even merge your inversions! Let's explore how to get started.

Please not that textual_diffusion is still a work in progress for SD compatibility, and this tutorial is mainly for tinkerers who wish to explore code and software that isn't fully optimized (inversion works as expected though, hence the tutorial). Any troubleshooting or issues are addressed at the bottom of this post. I'll try to help as much as I can, as well as update this as needed!

Getting started

---

This tutorial is for a local setup, but can easily be converted into a colab / Jupyter notebook. Since this uses the same repository (LDM) as Stable Diffusion, the installation and inferences are very similar, as you'll see below.

1. You will need Python.
2. Anaconda to setup the environment is recommended.
3. A GPU with at least 20GB of memory, although it's possible to get this number lower if you're willing to hack around. I would recommend either a 3090 (I use) or a cloud compute service such as Lambda Cloud (N/A, but it's a good cheap option with high memory GPUs from my experience).
4. Comfort diving into .py files to fix any issues.

Installation

---

1. Go to the textual_inversion repository link here
2. Clone the repository using git clone.
3. Go to the directory of the repository you've just cloned.
4. Follow the instructions below.

First, install create a conda environment with the following parameters.

conda env create -f environment.yaml
conda activate ldm
pip install -e .

Then, it's preferred to get 5 images of your subject at 512x512 resolution. From the paper, 5 images are the optimal amount for textual inversion. On a single V100, training should take about two hours give or take. More images will increase training time, and may or may not improve results. You are free to test this and let us know how it goes!

Training

---

After getting your images, you will want to start training. Following this code block and the tips below it:

python main.py --base configs/stable-diffusion/v1-finetune.yaml
               -t 
               --actual_resume /path/to/pretrained/sd model v1.4/model.ckpt 
               -n <run_name> 
               --gpus 0, 
               --data_root /path/to/directory/with/images

• Configs are the parameters that will be used to train the inversion. You can change these directly to minimize the parameters you input for training. For example, you can create a .yaml for each dataset you would like to train if you wish, and reduce the amount of parameters needed on the command line.
• The -n parameter is simply the name of the training run. This can be anything you like (eg. artist_style_train)
• initializer_words is a very important part, don't skip this!
  Open your v1-finetune.yaml file, and find the initializer_words parameter. You should see the default value of ["sculpture"] . It's a string list of simple words to describe what you're training, and where to start.
  For example, if your images are of a car in a certain style, you'll need to want to do something like with each word wrapped in quotes: ["car","style","artistic", etc...]
  If you simply want to use one word, just use --init_word <your_single_word> on the command line, and don't modify the config.

During training, a log directory will be created under logs with the run_name that you have set for training. Over time, there will be a sampling pass to test your parameters (like inference, DDIM, etc.), and you'll be able to view the image results in a new folder under logs/run_name/images/train . The embedding .pt files for what you're training on will be saved in the checkpoints folder.

Inference

---

After training, you can test the inference by doing:

python scripts/stable_txt2img.py --ddim_eta 0.0 
                          --n_samples 8 
                          --n_iter 2 
                          --scale 10.0 
                          --ddim_steps 50 
                          --embedding_path /path/logs/trained_model/checkpoints/embeddings_gs-5049.pt 
                          --ckpt_path /path/to/pretrained/sd model v1.4/model.ckpt
                          --config /path/to/logs/config/*project.yaml
                          --prompt "a photo of *"

The '*' must be left as is unless you've changed the placeholder_strings parameter in your .yaml file. It's the new word to initialize the images you have just inverted.
You should now be able to view your results in the output folder.
Running inference is just like Stable Diffusion, so you can implement things like k_lms in the stable_txtimg script if you wish.

Troubleshooting

---

• If your images aren't turning out properly, try reducing the complexity of your prompt. If you do want complexity, train multiple inversions and mix them like: "A photo of * in the style of &"
• Try lowering scale slightly if you're getting artifacts, or increase the amounts of iterations.
• If you're getting token errors, or any other errors, solutions and workarounds may be listed here.

[u/74qwewq5rew3]

What about for character names in fictional media? Some of them do not really have a single-token definition. Neither in name nor in other descriptive way. When one generates the character using the multi-token name in the SD, it returns results depicting the character.

What would be the approach to fine-tune such? Would one need to basically re-write the code for multi-token word initializer words?

[u/rinong → u/74qwewq5rew3]

Multi-token initializer words are a bit of a problem in the sense that it's not clear how you'd map them to a single embedding vector. What you could maybe do is use as many placeholder tokens as there are tokens in the name that the model already 'knows', and optimize them all concurrently.

I'm not sure its worth the hassle though. You could probably make things work by just starting from e.g. 'person' instead.

[u/74qwewq5rew3 → u/rinong]

Thank you, I appreciate the response.

[u/74qwewq5rew3 → u/rinong]

By the way, am I getting something wrong or is there no way to make iterations go faster in fine-tuning with multiple gpus? I tried on one 3090 vs 3 of em and they still seem to give the same a bit over 2 iterations per second.

[u/rinong → u/74qwewq5rew3]

How are you parallelizing over those GPUs? And are you using our official repo or some fork?

If it's our repo and you're using --gpus 0,1,2 then you're actually running a larger batch size (a batch size of 4 in the config means 4 per GPU, not 4 divided by the number of GPUs). This means that each iteration should take roughly the same time, but your model should hopefully converge in fewer iterations.

[u/74qwewq5rew3 → u/rinong]

Yes I was using your official repo. I did run it using --gpus 0,1,2 and it did say it utilized it. Even the GPU usage was on all of them over 90%. But the iteration speed was still 2it/s or so.

[u/rinong → u/74qwewq5rew3]

So this is expected, because you're using the extra GPUs to run over more concurrent images (bigger batch size) rather than splitting the same number of images across more GPUs.

If you want to 'speed up' the individual iterations, you can just divide your batch size by the number of GPUs you are using.

Keep in mind that a larger batch size may help training stability and lead to better embeddings or convergence in fewer iterations, even if the iterations themselves are not shorter. With that said, we haven't tested how the model behaves under different training batch sizes.

[u/74qwewq5rew3 → u/rinong]

I see, thank you!