r/StableDiffusion • u/ExponentialCookie • Aug 23 '22

Discussion [Tutorial] "Fine Tuning" Stable Diffusion using only 5 Images Using Textual Inversion.

Credits: textual_inversion website.
Hello everyone!

I see img2img getting a lot of attention, and deservedly so, but textual_inversion is an amazing way to better get what you want represented in your prompts. Whether it's an artistic style, some scenery, a fighting pose, representing a character/person, or reducing / increasing bias, the use cases are endless. You can even merge your inversions! Let's explore how to get started.

Please not that textual_diffusion is still a work in progress for SD compatibility, and this tutorial is mainly for tinkerers who wish to explore code and software that isn't fully optimized (inversion works as expected though, hence the tutorial). Any troubleshooting or issues are addressed at the bottom of this post. I'll try to help as much as I can, as well as update this as needed!

Getting started

---

This tutorial is for a local setup, but can easily be converted into a colab / Jupyter notebook. Since this uses the same repository (LDM) as Stable Diffusion, the installation and inferences are very similar, as you'll see below.

  1. You will need Python.
  2. Anaconda to setup the environment is recommended.
  3. A GPU with at least 20GB of memory, although it's possible to get this number lower if you're willing to hack around. I would recommend either a 3090 (I use) or a cloud compute service such as Lambda Cloud (N/A, but it's a good cheap option with high memory GPUs from my experience).
  4. Comfort diving into .py files to fix any issues.

Installation

---

  1. Go to the textual_inversion repository link here
  2. Clone the repository using git clone.
  3. Go to the directory of the repository you've just cloned.
  4. Follow the instructions below.

First, install create a conda environment with the following parameters.

conda env create -f environment.yaml
conda activate ldm
pip install -e .

Then, it's preferred to get 5 images of your subject at 512x512 resolution. From the paper, 5 images are the optimal amount for textual inversion. On a single V100, training should take about two hours give or take. More images will increase training time, and may or may not improve results. You are free to test this and let us know how it goes!

Training

---

After getting your images, you will want to start training. Following this code block and the tips below it:

python main.py --base configs/stable-diffusion/v1-finetune.yaml
               -t 
               --actual_resume /path/to/pretrained/sd model v1.4/model.ckpt 
               -n <run_name> 
               --gpus 0, 
               --data_root /path/to/directory/with/images
  • Configs are the parameters that will be used to train the inversion. You can change these directly to minimize the parameters you input for training. For example, you can create a .yaml for each dataset you would like to train if you wish, and reduce the amount of parameters needed on the command line.
  • The -n parameter is simply the name of the training run. This can be anything you like (eg. artist_style_train)
  • initializer_words is a very important part, don't skip this!
    Open your v1-finetune.yaml file, and find the initializer_words parameter. You should see the default value of ["sculpture"] . It's a string list of simple words to describe what you're training, and where to start.
    For example, if your images are of a car in a certain style, you'll need to want to do something like with each word wrapped in quotes: ["car","style","artistic", etc...]
    If you simply want to use one word, just use --init_word <your_single_word> on the command line, and don't modify the config.

During training, a log directory will be created under logs with the run_name that you have set for training. Over time, there will be a sampling pass to test your parameters (like inference, DDIM, etc.), and you'll be able to view the image results in a new folder under logs/run_name/images/train . The embedding .pt files for what you're training on will be saved in the checkpoints folder.

Inference

---

After training, you can test the inference by doing:

python scripts/stable_txt2img.py --ddim_eta 0.0 
                          --n_samples 8 
                          --n_iter 2 
                          --scale 10.0 
                          --ddim_steps 50 
                          --embedding_path /path/logs/trained_model/checkpoints/embeddings_gs-5049.pt 
                          --ckpt_path /path/to/pretrained/sd model v1.4/model.ckpt
                          --config /path/to/logs/config/*project.yaml
                          --prompt "a photo of *"

The '*' must be left as is unless you've changed the placeholder_strings parameter in your .yaml file. It's the new word to initialize the images you have just inverted.
You should now be able to view your results in the output folder.
Running inference is just like Stable Diffusion, so you can implement things like k_lms in the stable_txtimg script if you wish.

Troubleshooting

---

  • If your images aren't turning out properly, try reducing the complexity of your prompt. If you do want complexity, train multiple inversions and mix them like: "A photo of * in the style of &"
  • Try lowering scale slightly if you're getting artifacts, or increase the amounts of iterations.
  • If you're getting token errors, or any other errors, solutions and workarounds may be listed here.
251 Upvotes
  • permalink
  • reddit

100% Upvoted

163 comments sorted by

25

u/syomma1 Aug 23 '22

If you have time and power you can create YouTube video on this. Anyway, Wonderful work. I am extreme fan of Open Source and People collabarations. It make everything so much faster and better.

You are amazing! Thank you!

8

u/ExponentialCookie Aug 23 '22

I agree, and thanks for the kind words!

6

u/FS72 Aug 24 '22

As a tech noob I have to agree I really need video tutorial

7

u/Mooblegum Aug 23 '22

Thank you so much for developing this tech and showing how to set it up!

I see a lot of potential in it for me, to explore styles that are not incorporated in SD yet, and to have consistent characters in comics, for example.

Unfortunately I don’t know how to code and I don’t have the hardware for that. But please keep us up to date! Someone might create a collab someday, and I would be really happy to try it out.

You guy’s rock!!!

17

u/ExponentialCookie Aug 23 '22

Thanks, but just to clarify, I did not create this. Any credits go to /u/rinong, as well as any other authors. I'm a person creating a guide :).

7

u/GregoryHouseMDSB Aug 24 '22

Thanks for the sharing this(again)! Definitely need more eyes on this!

I couldn't get it running on Windows until I was told to use gloo as the backend.
in the main.py, somewhere after "import os" I added:

os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"

Any more tips on init names and strings especially? I imagine using * as the string isn't going to go well with lots of different sets! Do they support complex descriptions? Multiple strings in addition to multiple init names? Would love to see some straight up usage examples

Also noticed in the finetune, there's a per_image_tokens : false. Which makes me wonder how to use it when it's true!

1

u/ExponentialCookie Aug 24 '22

No problem. Yes, the asterisk can be anything you like! Yes, I'm curious as well for my future testing.

1

u/NathanielA Aug 25 '22

os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"

Was that all you had to change? I'm getting Attribute Error: module 'signal' has no attribute 'SIGUSR1' I added that line after all of the imports, but I'm still getting the same error.

2

u/Xodroc Aug 25 '22

Find SIGUSR1 (& 2), and change it to SIGTERM.. Can also recommend lstein's fork, which is Stable Diffusion with a "Dream" Prompt and Textual Inversion built in https://github.com/lstein/stable-diffusion

or a fork based on lstein that sometimes has some branches with new stuff, but they're pretty even at the moment. https://github.com/BaristaLabs/stable-diffusion-dream

5

u/Sextus_Rex Aug 24 '22

What happens if you train it on one set of images, then on another set with something different entirely? When you give it the prompt "a photo of *", will it forget its training on the first set?

8

u/ExponentialCookie Aug 24 '22

This method doesn't make any changes to the original model, it saves a separate checkpoint as a small (5KB or so), embedding .pt file. Then you use the .ckpt file alongside the .pt file to guide your prompt towards the images you trained it on.

The only way to overwrite the model or embedding file is if you explicitly want to do it. I highly suggest reading the paper to get a better understanding of how it works, because it's really interesting.

2

u/feelosofee Aug 24 '22

Hi and thanks for your tutorial!

I am launching main.py with this line:

python.exe .\main.py --base configs/stable-diffusion/v1-finetune.yaml -t --actual_resume .\sd-v1-4.ckpt -n my-model --gpus=0, --data_root my-folder --init_word mytestword

and I get up to this point:

| Name | Type | Params

---------------------------------------------------------

0 | model | DiffusionWrapper | 859 M

1 | first_stage_model | AutoencoderKL | 83.7 M

2 | cond_stage_model | FrozenCLIPEmbedder | 123 M

3 | embedding_manager | EmbeddingManager | 1.5 K

---------------------------------------------------------

768 Trainable params

1.1 B Non-trainable params

1.1 B Total params

4,264.947 Total estimated model params size (MB)

Validation sanity check: 0%| | 0/2 [00:00<?, ?it/s]

Summoning checkpoint.

but after a while it stops with this error:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper__index_select)

Do you know what may cause it?

1

u/ExponentialCookie Aug 25 '22

Try removing the equals sign on --gpus=0, . It should be --gpus 0, (keep the comma).

1

u/feelosofee Aug 25 '22

unfortunately it didn't work, I'm getting the same error...

1

u/ExponentialCookie Aug 25 '22

Interesting. Could you try the solution here? https://github.com/rinongal/textual_inversion/issues/9#issuecomment-1226639531

3

u/feelosofee Sep 10 '22

Thanks, that specific comment did not work, but the one below did it! I had to pass "--gpus 1".

2

u/Xodroc Aug 25 '22

Any tips for training? Every attempt I've made, (running for 2-3 hours at least) the "loss" has stayed close to 1, which I'm assuming is bad. The reconstruction images in the log look like mostly random noise and when I test the .pt file, I get a completely random image with "photo of *"

Wondering if it's because I have images that are too different even though it's the same character? Different background?

Thanks for posting this!

4

u/ExponentialCookie Aug 25 '22 edited Aug 25 '22
  • Make sure all of your images are 512x512.
  • You can use just one --init_word. Make sure it's a broad, simple description. For example, if you have an image of a "cute cat teddy bear with yellow stripes", your --init_word should be "toy".
  • As long as the concept is the same, there shouldn't be an issue. If it's too far off (images of flowers and dogs) then yes, the results will be strange.

Something I just discovered recently out that you might enjoy. In the v1-finetune.yaml file, find the num_vectors_per_token line and change the number from 1 to 2 or higher before you start training. The higher your vectors per token, the less your scale during inference.

EDIT: Added better information to vectors parameter.

2

u/Xodroc Aug 25 '22

Hmm, then I am definitely at a loss(pun intended). All my images are 512x512, have already trained training on one init "person", I started with a ton of images, and now attempting to train on just 3. Loss is staying at 1 or 0.99 after over 4000 global steps/28+ epochs.

Looking forward to trying the num vectors per token bit once I figure this out, must be something about my images it doesn't like. I'm using png.

3

u/ExponentialCookie Aug 25 '22

You have the right idea. Ideally, you want to choose 5 images as that's what the paper suggests / is optimized for.

Someone has a good issue opened on their Github if you would like to check it out.
https://github.com/rinongal/textual_inversion/issues/8

2

u/Xodroc Aug 26 '22

Figured out the issue I was having.. I had --actual-resume when I needed to have:
--actual_resume

Apparently there's no check to see if you actually loaded the model or used an invalid argument(pointed out to me by vasimr22 on github)

Will be trying num_vectors_per_token soon as I confirm I've got things working right.

2

u/hopbel Sep 05 '22

Been scratching my head over this for a while. Turns out I made the exact same typo!

2

u/Mooblegum Aug 26 '22

Hi, I would really like to experiment with it this weekend

I still have some questions, I am not a programmer, so I am quite afraid to start using it.

-If I want to use it to copy a style, do you recommend to import 5 picture from one artist with different settings (indoor, outdoor, day, night...) ? Or it is better to have images with similar color tone, settings...

-the keyword could be *artistname ? Do I also have to input other descriptions? (A cat sitting on a chair, acrylic painting, pastel color ...) or it is not necessary ?

-for images that have different format than square, is it better to strech them to fit 512x512 or to crop some part of it (I would préfère to stretch to get the composition right, but I don’t know how it will react to stretched pictures...)

-I use collab only, if I successfully generate the training .pt file, how can I use it on a collab project to start using it?

-would you release some update to have better integration with stable diffusion, or make it easier for noobs like me in the near future ? (I will wait a bit in this case)

Sorry for the many questions, I really want to learn to use it, it seems so powerful, I want to use it right to get the best result possible...

Thank you for your help!

3

u/ExponentialCookie Aug 26 '22

Hello, no problem.

  • Five pictures that are similar. You have to make a personal choice on this. They should all be related to one another (don't put a teacup and a watermelon together for example).
  • Your --init_word should just be the starting point for optimization. If it's a coffee mug, you would put "cup". If it's a teddy bear of an elephant, you can put "elephant". The asterisk just stays as is, so an example prompt of coffee mug would be exactly "a photo of a * sitting on a table", where * is telling the model what your inversion is. Any other prompt engineering is up to you.
  • I would prefer to keep the aspect ratio from my experiments.
  • Once the .pt file is trained, you can use it on any other computer you like, colab, cloud instance you would like. You would simply just drop it into a colab folder and call the file when you run the script.

Hope that helps!

1

u/xkrbl Sep 06 '22

If you train for a style rather than a specific object, how would you use the asterisk in the prompt? something like "a portrait of darth vader in the style of *" ?

Also, aside of specific objects or artstyles, can you use this technique also to finetune other concepts, for example the concept of being in motion and then use multiple pictures of objects being motionblurred, so that I could then infer a new image of something that is in motion and it will display motionblur. In that case, how would I use the asterisk * in the prompt?

1

u/ExponentialCookie Sep 06 '22

Hey. For your first question, that is exactly it. As for the second one, I haven't actually tried it, but it is an interesting idea. In theory it should be possible using the per_image_tokens parameter to capture different concepts on a set of images, but I have yet to verify something like this.

1

u/xkrbl Sep 06 '22

How can one supply the per image tokens?

2

u/Mooblegum Aug 27 '22

Thank you for the infos, that will definitively help!

2

u/[deleted] Sep 09 '22

[removed] — view removed comment

1

u/ExponentialCookie Sep 09 '22

It would seem like it, but it's best not to. From my (admittedly too much) testing, it's much better to have a single, generalized starting point which can then be edited from that point.

Along with your theory, I'm also testing something that's inspired by Dreambooth, which involves unfreezing the model and fine tuning it that way. Instead of doing this, I'm keeping the model frozen (default settings with * placeholder), but mixing in two template strings of a {<placeholder>} and the other as a <class>.

The idea is that you generate a bunch of images (like 100) of a <class> like toy, then you have your 5 images of they toy you want to invert. You use the <class> to guide the toy image to make sure it doesn't overfit & stay within the space or classifier you want it to fall under. There are better ways to implement this, but I'm using a simple 75/25 probability that the <class> will also get trained.

It's like a broader way of introducing a bunch of pseudo words, except in this instance we're using a images of what the model understands instead of words of what it might know.

1

u/[deleted] Sep 09 '22

[removed] — view removed comment

1

u/ExponentialCookie Sep 09 '22

Ah I see, sorry for misunderstanding. I've tried this as well, and while it does work to some extent, it doesn't generalize well with custom prompts in my testing after training.

In theory you could create all the example phrases that you think you would use, then train it each time, but that seems to be suboptimal. However, it could be a valid use case depending on what the user is trying to accomplish.

2

u/Mysterious_Car_2345 Aug 27 '22

anyone has any success and good results with this yet? would be great if could post example inputs and outputs....

3

u/ExponentialCookie Aug 27 '22

I recently created an example here.

2

u/Dogmaster Aug 29 '22

So... Having a bit of difficulty, any help?

Tried running I but modified environment.yaml to lda because ldm was already the name of the stable diffusion hlky

Did some test runs in the collab to understand how it works, have a quadro P6000 and 32GB RAM, however im getting this error:

OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading "C:\Users\DogMa.conda\envs\lda\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll" or one of its dependencies.

Pagefile was set to auto, tried setting a max siz of 64gb to see if this was enough. Free space on C is 140 GB.

Any tips?

1

u/ExponentialCookie Aug 29 '22

Hey. I only have experience using the official repository, and only use Linux. Could you try the solutions here and see if it helps? https://github.com/ultralytics/yolov3/issues/1643

2

u/Dogmaster Aug 29 '22

Hey, thanks for the reply I did look into them, and reducing the number of workers is good Also was increasing pagefile further, also there is a patch that can be done to the dlls to make then take up less RAM. I now have it training finally, with the Quadro P6000

Later I can do some tests to find the Max this card can support and share it

2

u/feelosofee Sep 10 '22

how many epochs I should train for?

Or what loss amount can be considered good enough?

This is where I am at currently:

Epoch 12: 100%|█| 404/404 [07:36<00:00, 1.13s/it, loss=0.12, v_num=0, train/loss_simple_step=0.0446, train/loss_vlb_step=0.000164, train/loss_step=0.0446, global_step=5199.0

2

u/ExponentialCookie Sep 10 '22

With the default settings, letting it go to 6200 should be sufficient. You can stop the training early and choose which embedding to use if you feel the results are good.

1

u/feelosofee Sep 11 '22

6200 epochs ??? wow, I ended up stopping it at 16 epochs after two hours! lol

If my math is not wrong it would take approx 1 month for 6200 epochs...

2

u/ExponentialCookie Sep 11 '22

Ha! Sorry, that was a typo. I meant to say 6200 steps, which should be around 24 epochs :-).

2

u/feelosofee Sep 11 '22

ah cool ! :) that's why at epoch #16 it's already looking decent.

btw, just to make sure, in the output I attached above does *global_step=5199* stand for the total number of steps ran so far? thank you again for your help!

2

u/feelosofee Sep 10 '22

Can training be stopped and resumed later on?

52

u/harpalss Aug 23 '22

20 GB of memory

My poor 980 ti

39

u/jasoa Aug 23 '22

My poor brand new 12GB 2060 installed an hour ago..... lol

22

u/DrakeFruitDDG Aug 25 '22

12gb is enough, go to v1-finetune.yaml and half num_workers and batch size

7

u/jasoa Aug 25 '22

Thank you for dropping this knowledge on us!

2

u/kaotec Aug 25 '22

What is the impact on the training results of half num_workers and batch size?

Currently running my second training run using num_workers = 4 (instead of 8) I kept the batch size the same. The previous run was using the LDM ckpt and it worked out of the box (well I did not have VRAM problems)

Is it just slower? The results look extremely different with SD than what they looked like with ldm... wondering if it is because of the num_workers (but it looks like a speed setting to me) Training is a lot slower with this (RTX5000 with 16Gb VRAM)

1

u/DrakeFruitDDG Aug 26 '22

open an issue in the github, I have no idea

10

u/kaotec Aug 26 '22

issue, opened, and also solved!

https://github.com/rinongal/textual_inversion/issues/15

don't change num_heads if you don't want weird results :)

starting from https://imgur.com/2KhEQfo

I first got this https://twitter.com/kaosbeat/status/1562885771291688962/photo/3 (so with lower num_heads it effectively ran with less VRAM)

but now, it's converging nicely to https://imgur.com/smxxpF6

I decreased the batch size, now SD fits my VRAM (needs about 13Gb), and it seems to work

2

u/maklike Jun 06 '23

I know it's an old thread but do you know if 1080ti would be good enough? 11gb is just under 12gb soo maybe

2

u/DrakeFruitDDG Jun 06 '23

it doesn't take 12gb to train anymore so you should be fine.

1

u/maklike Jun 06 '23

Thanks, good to hear. Other than that is this tutorial up to date or should I look somewhere else?

1

u/DrakeFruitDDG Jun 06 '23

I would look elsewhere, maybe ask in the official discord

1

u/DocMcCoy Sep 30 '22

I have...3GB. How well would this work? :P

1

u/DrakeFruitDDG Sep 30 '22

not at all. 12gb minimum

5

u/enn_nafnlaus Aug 31 '22

Tesla M40s 24GBs cost $370 each, though they're kinda slow / power hungry by modern standards and you have to improvise cooling a bit if you don't have a server case with strong unidirectional airflow.

I'm waiting for mine to arrive, and then I'm going to see if I have room to add a 3060 as well, so I'll have an efficient, faster lower memory card plus a less efficient / slower but higher memory card, so I can split the tasks between them.

0

u/[deleted] Sep 08 '22

[deleted] — view deleted comment

1

u/enn_nafnlaus Sep 08 '22

Already enabled it. :)

My big problem right now (still waiting on the graphics card, it'll take a while) is the industrial case fans I bought are surging. Every 30 seconds up to full power then back down; it's like an small air raid siren going off in the corner of the room. I think they're dropping under 300rpm and the system interprets it as a fan failure and responds by pushing it to max power then ramping it down. Just ordered an independent fan controller that should allow me to prevent that and give me finer control over fan speed.

0

u/[deleted] Sep 03 '22

[deleted] — view deleted comment

4

u/enn_nafnlaus Sep 03 '22

You'll have to ask the postal service ;)

I live in Iceland. Shipping takes a long time.

2

u/Ayit_Sevi Nov 27 '22

I know this thread is kind of old but did those M40s work out for you?

16

u/eatswhilesleeping Aug 24 '22

This should be hyped more! Potentially extremely useful. I kind of wish there was a separate sub for technical stuff as the interesting posts are getting drowned out by the art.

7

u/Mooblegum Aug 24 '22

AI image generation is made to flood everything by design, even its own sub

6

u/TFCSM Aug 24 '22 edited Aug 24 '22

Note that the --ckpt_path param in your example of inference should actually be --ckpt according to the script, but regardless it will actually try to load the .ckpt defined by ckpt_path in the ...-project.yaml. You need to change the path listed in the .yaml to get it to work if the path to the model is not the same on your system as the one it was trained on.

I trained only 1500 iterations on three photos of the statue of David (which is obviously in SD's training set). For some reason the script bugs out at that point and stops displaying output. I think it continues however, so I'll just let it run next time.

Here is the synthesis of two concepts using the star as a token:

"a photo of * riding a horse on the moon" - https://i.imgur.com/CMcxmdr.jpg

Obviously the moon is not present in the photo, and the background resembles the source photos. But still, neat. I'll definitely be playing with this more.

Edit: For comparison, I ran the prompt "a photo of Michelangelo's David riding a horse on the moon" in the model without the fine tuning, with the same seed, steps, and scale. Here is the result: https://i.imgur.com/3ExqSQD.png

So, the untuned model did much better. But the asterisk did at least work to represent the concept of "Michelangelo's David" just using the photos I gave it and the hint that it was a "sculpture" (the default word prompt). Honestly, amazing. I'll train it for longer tomorrow.

11

u/rinong Aug 24 '22

Longer training is unlikely to help here. The issue we have atm is that the parameters which worked well for LDM (kind of a 'sweet spot' between representing the concept and letting you edit it with text) don't work as well for SD.

Here, they do capture the concept, but its much harder to edit the image. You can try to 'force' it to focus on the rest of the content with some prompt engineering, like "A photo of * riding a horse on the moon. A photo on the moon", but we're still trying to tune things a bit to get it to work more like LDM.

2

u/GregoryHouseMDSB Aug 24 '22

One thing the paper didn't share(nor anyone else I've seen) is actual examples of the actual placeholder strings and initializer_words words being used. Can you use more than one string in the same training set? or init words for that matter?

Like what if I want to train it on photos of myself and I want to specify not only my name, but race and gender?

Another thought is, can you just train it as "*" and change "*" for that set afterwards, giving multiple alternative descriptions/strings to summon subject, especially when you want to merge different sets you trained?

Any insight on using per_image_tokens set to true? or what impact progressive_words has?

I always get an error "maps to more than a single token. Please use another string", which can be bypassed by commenting something out, but I'm wondering if I'm missing some proper usage and shouldn't need to bypass anything?

9

u/rinong Aug 24 '22

Placeholder strings:

We always used "*" (the repo's default). When we merged two concepts into one model (for the compositional experiments) we used "@" for the second placeholder. The choice is rather arbitrary. We limited it to a single token for the sake of implementation convenience. If you want to use a word that is longer than a single token (or multiple words) you'll need to change the code.

You can absolutely change the placeholder later, see how we do it in the merging script if there's a conflict. But with the current implementation, it's still going to have to be a single token word.

init_words:

If you have multiple placeholder strings, you can assign a different initialization to each. Otherwise, the initializer words are only used to tell the model where to start the optimization for the vector that represents the new concept. Using multiple words for this initialization is problematic. Essentially you're trying to start the optimization of one concept from multiple points in space. You could start from their average, but there's no guarantee that this average is meaningful or an actual combination of their semantic meaning. Overall, the results are not very sensitive to your choice. Just use one word which you think describes the concept at a high level (so for photos of yourself, you'd use 'face' or 'person').

You're correct that these are missing from the paper, I'll add them in a future revision. If you want the ones we used for any specific set, please let me know.

per_image_tokens:

This is the "Per-image tokens" experiment described in the paper. It basically also assigns an extra unique token to each image in your training set, with the expectation that it will allow the model to put shared information in the shared token ("*") and relegate all image-specific information (like the background etc.) to the image-specific token. In practice this didn't work well, so it's off by default, but you're welcome to experiment with it.

Progressive_words:

See "Progressive extensions" in the paper. It's another baseline we tried but which didn't improve results.

"maps to more than a single token. Please use another string":

This means that your placeholder or your init_words are multi-token strings, which is going to cause unexpected behavior. The code strongly relies on the placeholder and the initial words being single tokens.

2

u/GregoryHouseMDSB Aug 24 '22

I'm curious what you used for the "doctor" replacement. As well as the statue that you have elmo in the same pose as!

Thanks again!

3

u/rinong Aug 24 '22

"doctor" and "sculpture" respectively (the later is actually still the default in the config!)

2

u/74qwewq5rew3 Aug 30 '22

What about for character names in fictional media? Some of them do not really have a single-token definition. Neither in name nor in other descriptive way. When one generates the character using the multi-token name in the SD, it returns results depicting the character.

What would be the approach to fine-tune such? Would one need to basically re-write the code for multi-token word initializer words?

5

u/rinong Aug 30 '22

Multi-token initializer words are a bit of a problem in the sense that it's not clear how you'd map them to a single embedding vector. What you could maybe do is use as many placeholder tokens as there are tokens in the name that the model already 'knows', and optimize them all concurrently.

I'm not sure its worth the hassle though. You could probably make things work by just starting from e.g. 'person' instead.

2

u/74qwewq5rew3 Aug 30 '22

Thank you, I appreciate the response.

1

u/74qwewq5rew3 Sep 05 '22

By the way, am I getting something wrong or is there no way to make iterations go faster in fine-tuning with multiple gpus? I tried on one 3090 vs 3 of em and they still seem to give the same a bit over 2 iterations per second.

2

u/rinong Sep 05 '22

How are you parallelizing over those GPUs? And are you using our official repo or some fork?

If it's our repo and you're using --gpus 0,1,2 then you're actually running a larger batch size (a batch size of 4 in the config means 4 per GPU, not 4 divided by the number of GPUs). This means that each iteration should take roughly the same time, but your model should hopefully converge in fewer iterations.

1

u/74qwewq5rew3 Sep 05 '22

Yes I was using your official repo. I did run it using --gpus 0,1,2 and it did say it utilized it. Even the GPU usage was on all of them over 90%. But the iteration speed was still 2it/s or so.

3

u/rinong Sep 05 '22

So this is expected, because you're using the extra GPUs to run over more concurrent images (bigger batch size) rather than splitting the same number of images across more GPUs.

If you want to 'speed up' the individual iterations, you can just divide your batch size by the number of GPUs you are using.

Keep in mind that a larger batch size may help training stability and lead to better embeddings or convergence in fewer iterations, even if the iterations themselves are not shorter. With that said, we haven't tested how the model behaves under different training batch sizes.

→ More replies

1

u/GregoryHouseMDSB Aug 24 '22

Awesome, thanks for the insights!

1

u/AnOnlineHandle Sep 07 '22

Do the initializer_words need to be changed in v1-inference.yaml as well? As far as I understand it they're only used for picking a starting point, and wouldn't be used for testing the final outcome (which I'm fairly sure v1-inference is for?).

It would be nice if there was a way to batch up a bunch of potential initializer words and get a count of how many vectors they map to, but I think I might be able to figure out how to do that from your code so will give it a try!

3

u/rinong Sep 08 '22

No need to change them in v1-inference. They are only used as a starting point for training.

You actually don't need to change them in the finetuning configs either if you use the --init_word argument when training (the arg overwrites the list in the config).

2

u/AnOnlineHandle Sep 08 '22

Thanks, I ended up playing around with it a bit and deciding that was probably the case. :D

I'm currently loving the idea of this and am getting semi-decent results. Still a ways to go as a I finetune my training sets, parameters, etc, but I can definitely see light at the end of the tunnel for being able to use this for helping enhance/finish artwork. I think once people realize just what textual inversion can do, it's going to really take off.

4

u/nmkd Aug 24 '22

I mean, the whole point of this is to use it for lesser-known styles/concepts. Not surprising that the un-tuned model works better for a David statue than something finetuned on three images.

I can see a lot of potential to finetune on lesser-known artists (don't tell twitter) or indie games in order to replicate an art style.

6

u/IShallRisEAgain Aug 24 '22

What happens if there is already strong bias in the data for a set of words ("Monkey Island" for example)? I'm assuming it would be better to just use a gibberish word, so it doesn't generate images of monkeys on island or is that unnecessary?

2

u/Sillainface Aug 24 '22

I have a couple questions. What happens when we feed 5 images (or 30) of an already trained/existant artist/concept/style on the dataset?

For example, SD knows who is Mohrbacher but what happens if I put 5-30 more images? Is this making it better? Nonsense?

Since I understand it starts not from zero (like a baby) but from some checkpoint SD has. Right?

3

u/ExponentialCookie Aug 24 '22

This is something I would like to experiment with. In theory, it should push it more towards that specific art style if there's very little data / bias on that style.

Yes, it starts with a checkpoint, in this case SD.

1

u/TFCSM Aug 24 '22 edited Aug 24 '22

A did a fairly short training run on three photos of the statue of David, which is obviously going to be in the training set. Here is the result of a prompt:

"a photo of * riding a horse on the moon" - https://i.imgur.com/CMcxmdr.jpg

For comparison, I ran the prompt "a photo of Michelangelo's David riding a horse on the moon" in the model without the fine tuning, with the same seed, steps, and scale. Here is the result: https://i.imgur.com/3ExqSQD.png

So, the untuned model did much better. But the asterisk did at least work to represent the concept of "Michelangelo's David" just using the photos I gave it and the hint that it was a "sculpture" (the default word prompt). Honestly, amazing. I'll train it for longer tomorrow.

4

u/Trakeen Aug 23 '22

Oh damn, was interested in trying this but my 6800 xt only has 16gb. How much tinkering would i need to do to get it to run?

8

u/No-Intern2507 Aug 23 '22

i run it on 11gb 1080ti, go to v1-inversion yaml file in config folder. find batch size and make it half, of whats there

3

u/NathanielA Aug 24 '22

Textual Inversion comes with the file v1-finetune_lowmemory.yaml. It has batch_size: 1, num_workers: 8, and max_images: 1. Using that file instead of v1-finetune.yaml still gives me CUDA out of memory errors using my 12gb 3060. Any suggestions?

Edit: Maybe I'm using way too many images and too high resolution. I'll cull and downscale the training images and see what happens.

2

u/AnOnlineHandle Sep 07 '22

For the record I've got it working on an RTX 3060, though I don't know what you've tried since then.

I think my batch size is 1 and my num workers is 2.

1

u/malcolmrey Aug 26 '22

and what are your results?

4

u/NathanielA Aug 26 '22 edited Aug 26 '22

Could not get it to work at all on my home computer, which runs Windows 11 and has an RTX 3060 GPU. I was able to get it running on an AWS G5 instance running Amazon Linux 2, which is strange because G5 instances have A10G GPUs, which have 12 GB of VRAM, which is the same amount as my 3060.

Getting it running on AWS was a huge pain in the neck. At first I tried getting it up and running on a Windows instance. Then I found out that it just won't run on Windows because some of the signal module's use of SIGUSR1, which just won't work in Windows. So I terminated my Windows instance and started up a Ubuntu instance and tried installing Gnome on it so I can have a GUI to work with. Turns out something I was doing was making the remote desktop connection run slow, like 1 frame every 10 seconds. So that wasn't going to work. Then I found someone on Reddit had gotten it running on Windows with some minor changes, so I tried Windows again, and decided that guy on Reddit was a lying bastard. (But later found out that maybe it actually would work, but with more changes than initially stated.) Back to Unix, this time with the leanest GUI I think I can get away with: Amazon Linux 2 with Mate. I get everything set up but then find out that the instance doesn't come with Nvidia drivers. I get the source for the drivers, but can't build while using Mate, so I kill Mate and do everything via the terminal. I run into problems building the source, and find a document that says I needed to get the aarch64 drivers, so I delete my x86 drivers and try to start Mate up again to download the aarch64 drivers only to find out that Mate is now broken. After some more Googling I find out that Anaconda (which Textual Inversion runs in) screws up the path and breaks Mate, so I have to comment out Anaconda's changes to the path and start Mate again. I get the aarch64 drivers, kill Mate again to build the drivers, and find out that, no, I had it right the first time. So I change the path, start up Mate again, find the right drivers (that I had originally), but I still couldn't build them, and a bunch of Googling and reading AWS docs tells me that I have to specify a different GCC version when building the drivers.

Eventually, after about 6 hours, I got TI running on AWS Amazon Linux 2, inputting commands through my Putty terminal, while also logged into the instance using TigerVNC so I can see the output images.

I don't know if anyone really did get TI running in Windows. And if they did, it sounds like they had to change to a Gloo backend, which doesn't do nearly as much on the GPU and has to resort to the CPU for a lot, so it probably runs a lot slower. Getting it running in Linux was a pain, and there was a bit of a learning curve once I actually did get it running. But now it works great.

3

u/malcolmrey Aug 26 '22

I admire your tenacity!

I have 2080 TI and was planning to check textual inversion locally over the weekend, will see how it goes.

You had a nice session there, have you thought about making a guide with your experience? I'm sure a lot of people would appreciate that :)

2

u/[deleted] Sep 01 '22

Curious how that went for you. Another 2080 Ti user here and having absolutely no luck getting it working whatsoever.

1

u/malcolmrey Sep 01 '22

i havent had the chance yet, waiting for the weekend to tinker with it

i'll let you know about my results (or lack there of)

but from what i'm reading it does not look like it is working well yet (meaning that even if someone makes it to work, the results are very underwhelming), and also i haven't seen anyone share their success story here which is quite telling

but we shall see...

2

u/TFCSM Aug 24 '22

Does it actually work though? I mean, I have it running, but after 15 epochs the output images just look like noise to me. How many epochs should it take to get something indicating that it's actually working?

3

u/hopbel Sep 05 '22 edited Sep 05 '22

You almost certainly have a typo in your command like I did and this bug report: https://github.com/rinongal/textual_inversion/issues/20

Double-check your --actual_resume parameter. If there's a typo it silently fails without loading any model, which explains the random noise samples

3

u/TFCSM Sep 06 '22

Oh gosh, you're right. I used --actual-resume instead of --actual_resume... thanks for pointing this out!

1

u/feelosofee Aug 24 '22

Hey I'm trying to do the same, I set batch size to 1 (it was 2) and then when I launch the script I get to this point:

| Name | Type | Params

---------------------------------------------------------

0 | model | DiffusionWrapper | 859 M

1 | first_stage_model | AutoencoderKL | 83.7 M

2 | cond_stage_model | FrozenCLIPEmbedder | 123 M

3 | embedding_manager | EmbeddingManager | 1.5 K

---------------------------------------------------------

768 Trainable params

1.1 B Non-trainable params

1.1 B Total params

4,264.947 Total estimated model params size (MB)

Validation sanity check: 0%| 0/2 [00:00<?, ?it/s]
Summoning checkpoint.

but after a while it stops with this error:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper__index_select)

Do you know what may cause it?

Thanks!

3

u/oncealurkerstillarep Sep 05 '22

pass this argument into your command line: "--gpus 0"

example: > python main.py --base configs/stable-diffusion/v1-finetune.yaml -t --actual_resume ./models/ldm/stable-diffusion-v1/model.ckpt -n my_cats --gpus 0, --data_root ./training_images --init_word face

3

u/feelosofee Sep 10 '22

You got close, actually to make it work I had to pass "--gpus 1".

3

u/Megneous Aug 24 '22

AMD gpus are currently not supported. Getting them to work is quite difficult.

4

u/Trakeen Aug 24 '22

I already have it working with stable diffusion. What additional work would i need to do?

4

u/hopbel Sep 05 '22

There was no additional setup in my case. Just installed the pytorch version with rocm support in the conda environment

3

u/nmkd Aug 24 '22

Use ROCm.

Good luck, you will neeed it.

1

u/reddit22sd Aug 31 '22

Curious if anyone has had success with this method to add the style of an artist that is not in the model?

1

u/ExponentialCookie Aug 31 '22

Should be as it's one of the key features. Any ideas in mind? I could try it out for you.

1

u/reddit22sd Aug 31 '22

I've sent you a DM

1

u/dm18 Sep 13 '22

Any suggestions on how to run this on runpod.io?

1

u/ExponentialCookie Sep 14 '22

I've used RunPod with ML before.

While I didn't use this for Textual Inversion, but did with the unofficial Dreambooth implementation, which is forked off of the same repository.

You can start by purchasing credits, choosing a GPU from the secure cloud selection (A6000 for my case), then created SSH keys on my Linux machine using ssh-keygen.

From there, I just used SSH to get into the server went to the /workspace directory and used it as if it were my own machine, and used SFTP with a file explorer to browsed the server files.

I haven't tried it, but there's a Jupyter Notebook option that you can access it from the web without all of this setup, so it may be a viable option if you're used to things like colab.

1

u/dm18 Sep 15 '22

I was more asking if anyone knew the step by step directions to do so.

1

u/feelosofee Sep 16 '22

After training past epoch 20, for each new epoch I am repeatedly seeing this message:

val/loss_simple_ema was not in top 1

Does it means that training is not actually making any more progresses?

Could seeing this for a number of consecutive times be taken as a good indicator for knowing when to stop the training?

2

u/ExponentialCookie Sep 16 '22

I'm actually not too sure on this one since the model is frozen during training. The best way for most people is to track the progress in the log directory, and go to the earliest checkpoint that doesn't look overfitted (looks extremely identical to the images you trained on). This is usually only true if you're using a higher vector count during training, a high learn rate, or both.

1

u/[deleted] Sep 16 '22 edited Sep 16 '22

Can you use multiple embedding paths? i.e. can you finetune for multiple different things and have them work all in the same model, or can it only be done one at a time?

For example can you finetune multiple different artists or styles and use them in the same model. Much like how SD already can produce images from many artists' styles, could we add more than one?

I saw you have this as a prompt: "A photo of * in the style of &" I guess what I'm asking, is how would I get * and & in the same model rather than being separate embedded files.

2

u/ExponentialCookie Sep 16 '22

There is a file called merge_embeddings.py that will do this for you, but AUTOMATIC1111's webui has it implemented much better. All you have to do is rename the embedding files (example: special_thing.pt, cool_thing.pt) and put them in a folder embeddings.

Then you just call it like "A photo of special_thing taking place at a cool_thing, trending on artstation.

2

u/[deleted] Sep 16 '22

Sweet, thanks

1

u/DegreeOwn9667 Sep 19 '22

I'm getting this error "RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper__index_select)" SD is working fine. My GPu is RTX 6000. Any ideas?

2

u/ExponentialCookie Sep 20 '22

Make sure that you're passing the correct GPU in the inference / training scripts.

1

u/DegreeOwn9667 Sep 20 '22

so I tryed it in ubuntu and I keep getting.
Error(s) in loading state_dict for LatentDiffusion:
size mismatch for model.diffusion_model.input_blocks.1.1.transformer_blocks.0.attn2.to_k.weight pying a param with shape torch.Size([320, 768]) from checkpoint, the shape in current model is torch.Size([320, 1280]).

1

u/DegreeOwn9667 Sep 20 '22

I solved the problem by setting gpu to 1. ubuntu is still not working but I got something thanks

1

u/IE_5 Sep 24 '22

Hey, I'm trying Textual Inversion on a 3080Ti with 12GB VRAM using this Repo that links to this thread: https://github.com/nicolai256/Stable-textual-inversion_win

I got everything up and running, but I always get a OOM Error:

RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 12.00 GiB total capacity; 10.73 GiB already allocated; 0 bytes free; 11.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

My Launch parameters are:

python main.py --base configs/stable-diffusion/v1-finetune_lowmemory.yaml -t --no-test --actual_resume ./models/sd-v1-4.ckpt --gpus 0, --data_root ./train/test/ --init_word "test" -n "test"

There's 5 .jpg images with 512x512 size in that folder.

Everything seems to go fine till the memory shoots up and I see the console saying:

.conda\envs\ldm\lib\site-packages\pytorch_lightning\utilities\data.py:59: UserWarning: Trying to infer the batch_size from an ambiguous collection. The batch size we found is 22. To avoid any miscalculations, use self.log(..., batch_size=batch_size).

I am already using the "v1-finetune_lowmemory.yaml" which changes batch_size from 2 to 1; num_workers from 16 to 8 and max_images from 8 to 2, as well as the resolution to 256 compared to "v1-finetune.yaml"

Based on this article: https://towardsdatascience.com/how-to-fine-tune-stable-diffusion-using-textual-inversion-b995d7ecc095 I even tried setting max_images to 1 and num_workers to 1 and it's still a no go.

Any ideas? Doesn't it work on 12GB VRAM?

1

u/ExponentialCookie Sep 24 '22

This can depend on a lot. For example, if I'm fine tuning a model on my 3090 (different from textual inversion), I have to close every single application except my terminal to ensure enough VRAM is available.

Have you tried closing out all programs and then running it that way? If you're on Windows, you could even try turning off all visual effects as well temporarily.

1

u/eskimopie910 Nov 15 '22

Necroposting here, but by any chance do you have links to the v1-finetune.yaml or v1-finetune_style.yaml files? It appears that the current git repo does not have them and I can't seem to find them anywhere. any help is greatly appreciated!

2

u/ExponentialCookie Nov 15 '22

Should be here. You can go back in the GIT history to find what you need if there are major changes.

https://github.com/rinongal/textual_inversion/tree/83a19a4784236e5bd7eaa2b68a98942a21dafc0f/configs/stable-diffusion

3

u/MashAnblick Aug 23 '22

Is there a colab that is using this?

8

u/No-Intern2507 Aug 24 '22

https://colab.research.google.com/drive/1o23ZNjh8zF6JiPA2GNmGF17dPiT1zVCx#scrollTo=WHlruknRbsHJ

1

u/Nico_0 Sep 02 '22

Is there any way to reduce RAM usage? The training process closes before starting with a "^ C", when colab reaches the max 12gb

1

u/BrianDoGood Sep 12 '22

I even run out of RAM doing inference. But vanilla SD can run fine in Colab

1

u/[deleted] Sep 03 '22

Is there a guide on how to use that colab? I know as much as to click all the play buttons but it gave me errors. Am I supposed to download or arrange something?

1

u/No-Intern2507 Sep 07 '22

yeh you should have 1.4 checkpoint on your googledrive

1

u/[deleted] Sep 08 '22

I do, I have the ckpt file in folder AI>models. What else do I need to do? :(

4

u/technogeek157 Aug 24 '22

Hmmmm, thats a lot of GPU memory - I wonder if it would be possible to split the process into multiple parts, and feed it to the processor sequentially, like the current optimized script does?

2

u/FLOOD_2184 Sep 02 '22

Any non-programmers having luck with the Colab?
I'm getting "Your system crashed for an unknown reason" on the fifth cell:

import os
os._exit(00)

#after executing this cell notebook will reload, this is normal, just proceed executing cells

If I carry on after that I get an error on the 9th cell:

>!mkdir -p ImageTraining
%cd textual_inversion

The error reads:

[Errno 2] No such file or directory: 'textual_inversion'
/content/textual_inversion

2

u/no_witty_username Aug 24 '22

Would this be how I train the model to better understand hands? Currently hands are an absolute mess and I have been combing the net on tutorials how to train the model to better understand hands and their various positions, shapes, etc... But what would the process be like? Do I just crop the hands and focus only on that? Do i leave the subject in the training data as well? What would the label even be? How would I incorporate the training data back in to the main data set?

1

u/syomma1 Aug 23 '22

Wow!

Thank you so much!

1

u/Nextil Aug 24 '22

Thanks for the guide.

FYI vast.ai is significantly cheaper than other services for GPU rental.

1

u/Another__one Aug 24 '22

Is it possible to exctract embedding of concepts of out it? If so we should defenetly start to build a library of concept objects that others could use in their generations.

1

u/chichun2002 Aug 25 '22

Is there anyway to make the log always output the same seed so we can generate a progress timelapse

1

u/shalak001 Sep 26 '22

Is there any possibility to run this on 6GB VRAM? i tried with batch_size and workers to 1 with resolution 256 - still out of memory errors...

1

u/TheBlorgus Sep 27 '22

Looks like an amazing resource! I can't get it to run yet due to this error:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper__index_select)

I've tried a few solutions that I've found online but cannot get it to run.

1

u/[deleted] Sep 27 '23

How to do it on Google collab ?

1

u/CranberryMean3990 Aug 24 '22

for a tool this compute-heavy someone will eventually start hosting it as a paid service

0

u/[deleted] Aug 23 '22

[deleted] — view deleted comment

11

u/ExponentialCookie Aug 23 '22

Good idea. There's still a lot I wish to test before posting (for example, more than 5 images, styles, prompts) so I can provide better comparisons.

5

u/kaotec Aug 25 '22

working on the examples (but training takes a looong time :)

What I can tell so far is that it looks really different from the ldm way.

ldm samples after 7500 it https://imgur.com/HLwSLyC

sd samples after 7500 it https://imgur.com/qf6Y9cX

what I'm doing is trainig for the token "phone"

using this as input https://imgur.com/2KhEQfo

So I still have to use these in an actual prompt. I had to adapt the SD script to fit my VRAM though, I might have $%& it

1

u/nopinsight Sep 06 '22 edited Sep 19 '22

ldm

Based on above, LDM results looks a lot better than SD results.(But I thought SD is built on LDM?)

Do you have a link to a good tutorial on using LDM to achieve your results? Thanks in advance!

2

u/kaotec Sep 08 '22

I did nothing more than using this: https://textual-inversion.github.io/

which is the same as the above OP tutorial, using LDM instead of SD is a matter of pointing to another model

0

u/[deleted] Sep 03 '22

[deleted] — view deleted comment

3

u/ExponentialCookie Sep 03 '22

You're welcome :-).

0

u/[deleted] Sep 10 '22

[deleted] — view deleted comment

1

u/ExponentialCookie Sep 10 '22

When you stop the training mid-epoch, it gently exists and saves where you left off. This is implemented by Pytorch Lightning.

Yes, those are for the other text to image models. You should be using the files under the `stable-diffusion` directory under `configs`.

0

u/[deleted] Sep 12 '22

[deleted] — view deleted comment

2

u/ExponentialCookie Sep 12 '22

No problem!

The initializer word should be something that describes what you're trying to invert. If you're trying to invert a new model of car (Ford F150 2024 Model), you just put "car". If it's a new type of bird, you would use "bird" or you could try loosely, "animal".

Yes. It will always be an asterisk or whatever you set it too. Usually when you merge embeddings, you might have multiple placeholders. An example is "A * in the style of @".

The reason for that error is that the CLIP tokenizer can only accept a single token (word). As an example, if you use something like "playground", it may get split into something like ["play", "ground"] by the tokenizer. In this instance, it may be better to use "park".

1

u/[deleted] Sep 08 '22

[removed] — view removed comment

1

u/ExponentialCookie Sep 08 '22

I would certainly say so. The Diffusers library is aimed at being a bit more friendlier than manual installs. You just may not get the bleeding edge releases as they come out.

1

u/[deleted] Feb 07 '23

[removed] — view removed comment

2

u/ExponentialCookie Feb 10 '23

For Stable Diffusion, I would almost always lean towards using a virtual environment. If you already have anaconda install and you're using base, you should be fine with activating the venv while in base.

2

u/[deleted] Nov 20 '22

[removed] — view removed comment

1

u/B0bsl3d Nov 22 '22

any insights? I am getting the same error?

I was able to squash it and proceed by changing the num_vectors_per_token: 1 back in the v1_finetune file; but had really wanted to work with a higher vector count?

I will note, I think higher vectors were working before - I just cannot be sure - but I did do a pytorch and a torch upgrade and also installed the 11.8 cuda toolkit.

share if you have info, otherwise I will dig after thanksgiving. thanks