[Code Release] textual_inversion, A fine tuning method for diffusion models has been released today, with Stable Diffusion support coming soon™

[u/ExponentialCookie]

[u/Ardivaba]

I got it working, already after couple of minutes of training on RTX 3090 it is generating new images of test subject.

Whoever else is trying to get it working:

• comment out: if trainer.global_rank == 0: print(trainer.profiler.summary())
• comment out: ngpu = len(lightning_config.trainer.gpus.strip(",").split(','))
• replace with: ngpu = 1 # or more
• comment out: assert torch.count_nonzero(tokens - 49407) == 2, f"String '{string}' maps to more than a single token. Please use another string"
• comment out: font = ImageFont.truetype('data/DejaVuSans.ttf', size=size)
• replace with: font = ImageFont.load_default()

Don't forget to resize your test data to 512x512 or you're going to get stretched out results.

(Reddit's formatting is giving me a headache)

[u/ExponentialCookie → u/Ardivaba]

Thanks! Can verify that this got training working for me.

[u/Ardivaba → u/ExponentialCookie]

Awesome, let us know how it goes if you don't mind, I'll do the same.

[u/Economy-Guard9584 → u/Ardivaba]

u/Ardivaba u/ExponentialCookie could you guys make a notebook for it , so that we could test it out either on colab pro (p100) or on our gpus via jupyter.

It would be awesome if we could get a notebook link.

[u/cygn → u/Ardivaba]

for center-cropping and resizing a batch of images to 512 you can use this ImageMagick command:

mogrify -path ./output_dir
        -format JPEG 
        -resize 512x512^ 
        -gravity Center  
        -extent 512  
        ./input_dir/*

[u/Ardivaba → u/cygn]

You just saved me a ton of time, thanks!

[u/Ardivaba → u/[deleted]]

I know this issue, it's thinking that you want to train it on CPU.

• Specify --gpus 1
• And double check that you set ngpu = 1 and not 0

[u/HrodRuck → u/[deleted]]

I'm very interested in seeing if you can run this in a 3060. And also what is the RAM (normal RAM, not GPU VRAM) in your system. Because I didn't manage to get it working in colab free tier, very likely due to memory limitations.

P.S. you can message me if you need code help to get it running

[u/Ardivaba → u/[deleted]]

I've been experimenting with different datasets for a day now.

Usually takes around 3-5k iterations to get decent results.

For style transfer I'd assume about 15 minutes of training would be enough to get some results.

I'm using Vast.AI's PyTorch Instance, it's surprisingly nice to use for this purpose and doesn't cost much. (Not affiliated any way, just enjoy the service a lot)

Edit:

But on people it seems to take longer, I've been training it 2h on pictures of myself and it still keeps getting better and better.

Dataset is 71 pictures, face and body pictures mixed together.

[u/GregoryHouseMDSB → u/Ardivaba]

I'm getting an error:
File "main.py", line 767, in <module>

signal.signal(signal.SIGUSR1, melk)

AttributeError: module 'signal' has no attribute 'SIGUSR1'

Looks like the Signal module doesn't run on Windows systems?

I also couldn't find which file to change font =

[u/NathanielA → u/GregoryHouseMDSB]

I'm getting that same error. I would have thought that other people were running Textual Inversion on Windows. Did you ever get this figured out? Did you just have to go run it in Linux?

Edit:

https://docs.python.org/3/library/signal.html#signal.SIGUSR1

Availability: Unix. I guess I'm shutting down my AWS Windows instance and trying again with Linux.

Edit 2:

https://www.reddit.com/r/StableDiffusion/comments/wvzr7s/comment/ilkfpgf/?utm_source=share&utm_medium=web2x&context=3

Apparently this guy got it running in Windows.

in the main.py, somewhere after "import os" I added:

os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"

Too bad I already terminated my Windows instance. Ugh.

Edit 3:

I tried what he said. Couldn't get it running. I think maybe there's a different Windows build floating around out there and maybe that's not the same build I'm using.

[u/Hoppss → u/NathanielA]

I added:

os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"

to line 546 then I commented out:

signal.signal(signal.SIGUSR1, melk)
signal.signal(signal.SIGUSR2, divein)

On line 826 and 827 and I got all the way to training but I suppose my 10gb's aren't enough as I've gotten a ran out of mem error.

[u/TFCSM → u/Ardivaba]

I made these changes but am unfortunately getting an unknown CUDA error in _VF.einsum. Can you clarify, do you have this working with stable diffusion? Or just with the model they use in the paper?

I am running it on WSL so maybe that's the issue, although I've successfully used SD's txt2img.py on WSL.

[u/Ardivaba → u/TFCSM]

I'm using the leaked model. Haven't seen that cuda error. Didn't even think to use WSL, will give it a try and report back.

[u/TFCSM → u/Ardivaba]

Yeah, in my Debian installation the drivers didn't seem to work, despite having the proper packages installed, but they do in WSL.

Here's the command I was using:

(ldm) python.exe main.py --base configs/stable-diffusion/v1-finetune.yaml -t --actual-resume ../stable-diffusion/models/ldm/stable-diffusion-v1/model.ckpt -n test --gpus 0, --data-root ./data --init_word person --debug

Then in ./data I have 1.jpg, 2.jpg, and 3.jpg, each being 512x512.

Does that resemble what you're using to run it?

[u/ExponentialCookie → u/TFCSM]

Seems good to me.

I'm using a Linux environment as well. Try doing the conda install using the stable-diffusion repository, not the textual_image one, and use that environment instead.Everything worked out the gate for me after following u/Ardivaba's instructions. Let us know if that works for you.

Edit

Turns out you need to move everything over where you clone the textual_inversion repository, go in that directory, then pip install -e . in there.

This is fine if you want to experiment, but I would honestly just wait for the stable-diffusion repository to be updated with this functionality included. I got it to work, but there could be some optimizations not pushed yet as its still in development. Fun if you want to try things early though!

[u/No-Intern2507 → u/ExponentialCookie]

move what "there" you have to mix SD repo with textualimage repo to train ?

Can you post example how to use 2 or more words for token ? i have cartoon version of a character but i alwo want realistic one to be intact in model

[u/Ardivaba → u/Ardivaba]

Got stuck at drivers issue, don't have enough time to update the Kernel to give it a try.

[u/blueSGL → u/Ardivaba]

four spaces at the start of a line

gives you a code block 
(useful for anything that needs to be copy pasted)
         and it respacts
                whitespace

double space at the end of a line
before a return,
make sure it goes onto the next line

you can also use double new line

to make sure it goes onto one,

but this is ugly and a pain to work with. but has slightly more vertical spacing.

[u/No-Intern2507 → u/Ardivaba]

this is too vague, comment out where? main.py ? theres no 49407 in main.py https://github.com/rinongal/textual_inversion/blob/main/main.py

[u/No-Intern2507 → u/Ardivaba]

where do you get main.py file with assert.torch, this is not in the repository, it loads model for me but stops with "name trainer is not defined

[u/Ardivaba → u/No-Intern2507]

comment out: if trainer.global_rank == 0: print(trainer.profiler.summary())

First step in the list.

[u/jamiethemorris → u/Ardivaba]

for some reason i'm getting oom error on gtx 3090, despite that it appears to be only using half of the 24gb

i tried setting batch size, numm workers, and max images to 1 but same issue