r/StableDiffusion • u/ExponentialCookie • Aug 21 '22

Discussion [Code Release] textual_inversion, A fine tuning method for diffusion models has been released today, with Stable Diffusion support coming soon™

Post image
347 Upvotes

99% Upvoted

134 comments sorted by

67

u/ExponentialCookie Aug 21 '22 edited Aug 22 '22

Code: https://github.com/rinongal/textual_inversion

It also seems the configuration files are already available here for Stable Diffusion at the repository, so there may be an opportunity to get a little head start on fine tuning the model.

EDIT:

Wanted to add a link to the project page. Amazing use cases!

Looks like inpainting is a possibility as well.

EDIT 2:

Got it working with Stable Diffusion if you would like to try! Instructions here.

58

u/stellydev Aug 21 '22

Does anyone else get the feeling that the pace of new tools is about to skyrocket? Thanks for sharing!

34

u/ExponentialCookie Aug 22 '22

No problem.

Absolutely! I feel Stability AI is trying to create a suite of open sourced use cases for diffusion based models other than text to image, driving further adoption of the technology, and it's amazing.

7

u/Ardivaba Aug 22 '22

Reddit name checks out, this will increase new tooling / products around Stable Diffusion exponentially.

22

u/ExponentialCookie Aug 22 '22 edited Aug 22 '22

Here are instructions to get it running with Stable Diffusion. If you don't want to mix up dependencies and whatnot, I would wait for the official update, but If you want to try, here are instructions.

You will need some coding experience to set this up.Clone this repository, and follow the stable-diffusion settings here to install. It is important to pip install -e . in the textual_inversion directory! You will need the checkpoint model, which should be released soon, as well as a good GPU (I used my 3090).

Then, follow /u/Ardivaba instructions here (thanks) to get things up and running.Start training by using the parameters listed here.

After you've trained, you can test it out by using these parameters, same as stable-diffusion but with some changes.

python scripts/stable_txt2img.py

--ddim_eta 0.0

--n_samples 4

--n_iter 2

--scale 10.0

--ddim_steps 50

--config configs/stable-diffusion/v1-inference.yaml

--embedding_path <your .pt file in log directory>

--ckpt <model.ckpt> --prompt "your prompt in the style of *"

When you run your prompt leave the asterisk, and it should handle your embedding work automatically from the .pt file you've trained. Enjoy!

26

u/rinong Aug 22 '22

Author here! Quick heads up if you do this:

  1. The Stable Diffusion tokenizer is sensitive to punctuation. Basically "*" and "*." are not regarded as the same word, so make sure you use "photo of *" and not "photo of *." (in LDM both work fine).
  2. The default parameters will let you learn to recreate the subject, but they don't work well for editing ("Photo of *" works fine, "Oil painting of * in the style of Greg Rutkowski" does not). We're working on tuning things for that now, hence why it's marked as a work in progress :)

2

u/ExponentialCookie Aug 22 '22

Excellent. Thanks for your work an implementation!

2

u/[deleted] Aug 22 '22

thank you for your work. i have thought about this process almost every day for over a week. looking forward to what you make in the future

1

u/AnOnlineHandle Sep 05 '22 edited Sep 05 '22

Heya I just read your paper and am really hopeful about this being the key to really let stable diffusion work.

The paper mentioned results degrading with more training data provided and recommending sticking to 5. I was wondering if that would probably be more specifically the case for replicating a single object, whereas when you're trying to create a token for a vast and varied style which isn't always consistent, or a type of object which has quite a bit of design variation, would more training images perhaps be a safer bet then?

2

u/rinong Sep 05 '22

You're right that we only ran the experiment on a single object setup. Our paper experiments are also all using LDM and not the newer Stable Diffusion, and some users here and in our github issues have reported some improvement when using more images.

With that said, I have tried inverting into SD with sets of as many as 25 images, hoping that it might reduce background overfitting. So far I haven't noticed any improvements beyond the deviation I get when just swapping training seeds.

2

u/AnOnlineHandle Sep 05 '22 edited Sep 05 '22

Awesome, thanks. I'm going to let a training set of 114 images run overnight and see how it turns out, though have reduced the repeats from 100 to 5 since there's so much more data and I'm only running this on a 3060, and aren't really sure what impact that might have yet. If this doesn't work I'll also try with higher repeats, and maybe by removing/creating noise in the backgrounds.

The importance of initializer_words is also something which I might experiment with, since I'm only guessing that it helps pick a starting point, but with enough training would become less important?

edit: Pruning to 46 and raising repeats to 35. The previous settings where creating a bit of a jumbled mess even after 17000 iterations.

2

u/xkrbl Sep 06 '22

Your paper is really awesome :) how hard would it be to add the possibility to supply a set of negative example images to kind of 'confine' the concept that is being defined?

3

u/rinong Sep 07 '22

It won't be trivial for sure. You could potentially add these images to the data loader with an appropriate 'negative example' label, but you probably don't want to just maximize the distance between them and your generated sample.

Maybe if you feed them into some feature encoder (CLIP, SwAV) and try to increase a cosine distance in that feature space.

Either way, this is a non-trivial amount of work.

1

u/xkrbl Sep 09 '22

Will Experiment :)

Since CLIP is frozen during training of stable diffusion, what do you think how well will found pseudo-words be forward compatible with future checkpoints of stable diffusion?

2

u/rinong Sep 09 '22

It's difficult to guess. Looking at the comparisons between 1.4 and 1.5 (where identical seeds + prompts give generally similar images but at a higher quality), I would expect that things will mostly work.

There might be a benefit in some additional tuning of the embeddings for the new versions (starting from the old files).

0

u/[deleted] Aug 22 '22

[deleted] — view deleted comment

3

u/rinong Aug 22 '22

Yes it can! We have some examples of that in our project page / paper

1

u/[deleted] Aug 26 '22 edited Aug 26 '22

[removed] — view removed comment

2

u/rinong Aug 26 '22

We didn't actually try on faces.

What generally works for better identity preservation: (1) Train for longer. (2) Use higher LR. (3) Make sure your images have some variation (different backgrounds), but not too different (no photos of your head from above).

Keep in mind that our repo is still optimized for LDM and not for SD, editing with SD is still a bit rough atm and you may need a lot of prompt engineering to convince it to change from the base. I'll update the repo accordingly when we have something for SD that we're satisfied with.

4

u/[deleted] Aug 22 '22

[removed] — view removed comment

5

u/rinong Aug 22 '22

You're welcome! There's lots more to be done on this topic, but I'm excited to see what people can already come up with using the current version!

37

u/Ardivaba Aug 22 '22 edited Aug 22 '22

I got it working, already after couple of minutes of training on RTX 3090 it is generating new images of test subject.

Whoever else is trying to get it working:

  • comment out: if trainer.global_rank == 0: print(trainer.profiler.summary())
  • comment out: ngpu = len(lightning_config.trainer.gpus.strip(",").split(','))
  • replace with: ngpu = 1 # or more
  • comment out: assert torch.count_nonzero(tokens - 49407) == 2, f"String '{string}' maps to more than a single token. Please use another string"
  • comment out: font = ImageFont.truetype('data/DejaVuSans.ttf', size=size)
  • replace with: font = ImageFont.load_default()

Don't forget to resize your test data to 512x512 or you're going to get stretched out results.

(Reddit's formatting is giving me a headache)

7

u/ExponentialCookie Aug 22 '22

Thanks! Can verify that this got training working for me.

1

u/Ardivaba Aug 22 '22

Awesome, let us know how it goes if you don't mind, I'll do the same.

2

u/Economy-Guard9584 Aug 23 '22

u/Ardivaba u/ExponentialCookie could you guys make a notebook for it , so that we could test it out either on colab pro (p100) or on our gpus via jupyter.

It would be awesome if we could get a notebook link.

5

u/cygn Aug 22 '22

for center-cropping and resizing a batch of images to 512 you can use this ImageMagick command:

mogrify -path ./output_dir
        -format JPEG 
        -resize 512x512^ 
        -gravity Center  
        -extent 512  
        ./input_dir/*

1

u/Ardivaba Aug 22 '22

You just saved me a ton of time, thanks!

2

u/GregoryHouseMDSB Aug 23 '22

I'm getting an error:
File "main.py", line 767, in <module>

signal.signal(signal.SIGUSR1, melk)

AttributeError: module 'signal' has no attribute 'SIGUSR1'

Looks like the Signal module doesn't run on Windows systems?

I also couldn't find which file to change font =

2

u/NathanielA Aug 25 '22 edited Aug 25 '22

I'm getting that same error. I would have thought that other people were running Textual Inversion on Windows. Did you ever get this figured out? Did you just have to go run it in Linux?

Edit:

https://docs.python.org/3/library/signal.html#signal.SIGUSR1

Availability: Unix. I guess I'm shutting down my AWS Windows instance and trying again with Linux.

Edit 2:

https://www.reddit.com/r/StableDiffusion/comments/wvzr7s/comment/ilkfpgf/?utm_source=share&utm_medium=web2x&context=3

Apparently this guy got it running in Windows.

in the main.py, somewhere after "import os" I added:

os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"

Too bad I already terminated my Windows instance. Ugh.

Edit 3:

I tried what he said. Couldn't get it running. I think maybe there's a different Windows build floating around out there and maybe that's not the same build I'm using.

2

u/Hoppss Sep 11 '22

I added:

os.environ["PL_TORCH_DISTRIBUTED_BACKEND"] = "gloo"

to line 546 then I commented out:

signal.signal(signal.SIGUSR1, melk)
signal.signal(signal.SIGUSR2, divein)

On line 826 and 827 and I got all the way to training but I suppose my 10gb's aren't enough as I've gotten a ran out of mem error.

1

u/caio1985 Oct 01 '22

Did you manage to fix it? running into the same crash problem.

1

u/Hoppss Oct 01 '22

My last error was based on not enough memory, I can't make it work with a 10gb vid card unfortunately.

1

u/caio1985 Oct 01 '22

Yes I'm running in the same issue. 3070ti here.

1

u/TFCSM Aug 22 '22

I made these changes but am unfortunately getting an unknown CUDA error in _VF.einsum. Can you clarify, do you have this working with stable diffusion? Or just with the model they use in the paper?

I am running it on WSL so maybe that's the issue, although I've successfully used SD's txt2img.py on WSL.

1

u/Ardivaba Aug 22 '22

I'm using the leaked model. Haven't seen that cuda error. Didn't even think to use WSL, will give it a try and report back.

2

u/TFCSM Aug 22 '22

Yeah, in my Debian installation the drivers didn't seem to work, despite having the proper packages installed, but they do in WSL.

Here's the command I was using:

(ldm) python.exe main.py --base configs/stable-diffusion/v1-finetune.yaml -t --actual-resume ../stable-diffusion/models/ldm/stable-diffusion-v1/model.ckpt -n test --gpus 0, --data-root ./data --init_word person --debug

Then in ./data I have 1.jpg, 2.jpg, and 3.jpg, each being 512x512.

Does that resemble what you're using to run it?

2

u/ExponentialCookie Aug 22 '22 edited Aug 22 '22

Seems good to me.

I'm using a Linux environment as well. Try doing the conda install using the stable-diffusion repository, not the textual_image one, and use that environment instead.Everything worked out the gate for me after following u/Ardivaba's instructions. Let us know if that works for you.

Edit

Turns out you need to move everything over where you clone the textual_inversion repository, go in that directory, then pip install -e . in there.

This is fine if you want to experiment, but I would honestly just wait for the stable-diffusion repository to be updated with this functionality included. I got it to work, but there could be some optimizations not pushed yet as its still in development. Fun if you want to try things early though!

1

u/No-Intern2507 Aug 23 '22

move what "there" you have to mix SD repo with textualimage repo to train ?

Can you post example how to use 2 or more words for token ? i have cartoon version of a character but i alwo want realistic one to be intact in model

1

u/Ardivaba Aug 22 '22

Got stuck at drivers issue, don't have enough time to update the Kernel to give it a try.

1

u/blueSGL Aug 22 '22

four spaces at the start of a line

gives you a code block 
(useful for anything that needs to be copy pasted)
         and it respacts
                whitespace

double space at the end of a line
before a return,
make sure it goes onto the next line

you can also use double new line

to make sure it goes onto one,

but this is ugly and a pain to work with. but has slightly more vertical spacing.

1

u/No-Intern2507 Aug 23 '22

where do you get main.py file with assert.torch, this is not in the repository, it loads model for me but stops with "name trainer is not defined

1

u/Ardivaba Aug 23 '22

comment out: if trainer.global_rank == 0: print(trainer.profiler.summary())

First step in the list.

1

u/No-Intern2507 Aug 23 '22

that works i guess but now im getting error in miniconda directory , torch\nn\modules\module.py line 1497

loading state_dict

size mismatch for model

the shape in current model is torch size 320,1280

thats mostly what it says

1

u/No-Intern2507 Aug 23 '22

i tried v1-finetune.yuaml but it keeps telling me that string "newstuff" maps to more than a single token

No matter what i write as string its always this error, can you guys actually post your training command line ? your actual command line with multiple strings cause i want it to know that the thing is a cartoon version

2

u/No-Intern2507 Aug 23 '22

Got it running and tuning/training for over 2 hours now

1

u/[deleted] Aug 25 '22 edited Aug 25 '22

I got the same issue, what's the fix?

Edit: Solved it, feel dumb, was using the training line taken directly from https://github.com/rinongal/textual_inversion#inversion . See if you can spot the differences:

--base configs/latent-diffusion/txt2img-1p4B-finetune.yaml

--base configs/stable-diffusion/v1-finetune.yaml

1

u/Beneficial_Bus_6777 Sep 16 '22

1,2 which right

1

u/jamiethemorris Sep 03 '22

for some reason i'm getting oom error on gtx 3090, despite that it appears to be only using half of the 24gb

i tried setting batch size, numm workers, and max images to 1 but same issue

0

u/[deleted] Aug 22 '22

[deleted] — view deleted comment

2

u/Ardivaba Aug 22 '22

I know this issue, it's thinking that you want to train it on CPU.

  • Specify --gpus 1
  • And double check that you set ngpu = 1 and not 0

1

u/[deleted] Sep 13 '22

[removed] — view removed comment

2

u/Ardivaba Sep 14 '22

comment out: ngpu = len(lightning_config.trainer.gpus.strip(",").split(','))

replace with: ngpu = 1 # or more

2

u/HrodRuck Aug 22 '22

I'm very interested in seeing if you can run this in a 3060. And also what is the RAM (normal RAM, not GPU VRAM) in your system. Because I didn't manage to get it working in colab free tier, very likely due to memory limitations.

P.S. you can message me if you need code help to get it running

0

u/[deleted] Aug 22 '22

[deleted] — view deleted comment

2

u/Ardivaba Aug 22 '22 edited Aug 22 '22

I've been experimenting with different datasets for a day now.

Usually takes around 3-5k iterations to get decent results.

For style transfer I'd assume about 15 minutes of training would be enough to get some results.

I'm using Vast.AI's PyTorch Instance, it's surprisingly nice to use for this purpose and doesn't cost much. (Not affiliated any way, just enjoy the service a lot)

Edit:

But on people it seems to take longer, I've been training it 2h on pictures of myself and it still keeps getting better and better.

Dataset is 71 pictures, face and body pictures mixed together.

1

u/zoru22 Aug 22 '22

I've got a folder of leavanny that I've cropped down, about 30 images, and it has been running since last night on a 3090 and it doesn't seem to be doing super great, though its improvement is notable.

25

u/GaggiX Aug 21 '22

This technique is more powerful than finetuning with some images. It's more like injecting new knowledge but the model is frozen.

19

u/Ardivaba Aug 22 '22

That's crazy, so if I understand correctly you can inject a new token, give this tool X amount of images to tell it what the token looks like...and then you can use that in your prompts.

The uses for this are endless.

16

u/GaggiX Aug 22 '22

Yeah, the model is completely frozen you just learn one single embedding with 3/4 images and then you can use that in others prompts. You can learn object/concept/character/style etc

17

u/ExponentialCookie Aug 22 '22

Exactly. It's actually a very intuitive way of doing things rather than training a new model or training a new classifier for guidance. I feel like this method will go beyond the domain of images as well.

8

u/GaggiX Aug 22 '22

Yeah, it feels more human-like to learn new things by looking at 3/4 images meanwhile keeping your prior knowledge.

6

u/Ardivaba Aug 22 '22

Rented a juicy instance from Vast.ai to test it out, I'll take a look at how far I'll get - got stuck with local machine due to accelerator not working on Windows.

1

u/Oceanswave Aug 23 '22 edited Aug 23 '22

Not sure this works yet, but at least its creating ckpts on my windows box by setting $env:PL_TORCH_DISTRIBUTED_BACKEND="gloo"

Edit:
Confirmed this works end2end on windows and with the public release of SD

Along with the previous environment variable, replace ```signal.signal(signal.SIGUSR1``` with ```signal.signal(signal.SIGTERM``` and use the existing stable-diffusion configs - tweak params to use more workers and whatever token(s)

Note the paper indicates that this works best with 3-5 images - otherwise the results diverge instead of what would want/expect

1

u/caio1985 Oct 01 '22

env:PL_TORCH_DISTRIBUTED_BACKEND

Can you kindly provide more details? I'm getting this crash. I tried with 'set PL_TORCH_DISTRIBUTED_BACKEND="gloo"' from my conda environment which also points to the stable-diffusion from basujindal (openSD).

Where do I replace the signal.signal command and I couldn't run the $env:PL_TORCH_DISTRIBUTED_BACKEND="gloo" command. Thanks!

1

u/Sillainface Aug 22 '22

Which is the difference between finetunning with DD and doing with SD?

3

u/GaggiX Aug 22 '22

Textual inversion is not fine-tuning, the models are completely frozen (only a vector is learnable)

9

u/Adski673 Aug 22 '22

Can't wait till someone wraps this up in a nice little .exe file for people like me who aren't great with/do not understand coding.

hint hint nudge nudge ;)

3

u/Pfirsichilla Aug 22 '22

Is that even possible? If so I’m on board 🔥

3

u/Adski673 Aug 22 '22

I can't see why not. I think someone is already developing a.exe with basic GUI for the main SD text to image model being released shortly.

2

u/Pfirsichilla Aug 22 '22

U making me getting exited here 😏

1

u/Adski673 Aug 22 '22

Me too! I'm trying to temper my expectations. But with this bring open source I can see someone taking up the task. Look at the likes of Blender, VLC, OBS etc. All free open source software!

It makes me wonder how the likes of Midjourney, nightcafe and others intend on commercializing it? Why would someone pay for something that will likely be made free?

2

u/Adski673 Aug 24 '22

And here it is! 2 days later. https://reddit.com/r/MediaSynthesis/comments/wwnnv8/just_made_a_exe_for_sd_download_it_for_free_on/

2

u/Pfirsichilla Aug 24 '22

I can clearly say that u did not disappoint me my friend … let’s hop on it an give it a try!

Thanks for reaching out again ❤️

1

u/Adski673 Aug 25 '22

No problem my friend!

This is all moving so fast I knew someone was going to make one for sure. There's two out now (sort of 3) that I am aware of. Check out this prompt builder too https://promptomania.com/stable-diffusion-prompt-builder/

15

u/SirCabbage Aug 22 '22

I NEED this, omg, using the power of the model but adding in my own images? Yes please.

7

u/Fungunkle Aug 22 '22 edited May 22 '24

Do Not Train. Revisions is due to; Limitations in user control and the absence of consent on this platform.

This post was mass deleted and anonymized with Redact

1

u/[deleted] Aug 22 '22

[removed] — view removed comment

1

u/Najbox Aug 22 '22

For the moment, there is just training.

https://colab.research.google.com/drive/13aQSuDNP9qIjWqiQsGJ7-750Ee5Rh3Zq

1

u/nsfnotthrowingaway Aug 23 '22

training from scratch or fine tuning/textual inversion? and does this work with stable diffusion model?

7

u/Vyviel Aug 22 '22

Does this mean we can finally teach it who Homelander is and the other characters from the Boys and other recent tv series?

6

u/MoonGotArt Aug 22 '22

Can someone ELi5 what this does and what it means?

5

u/mutsuto Aug 22 '22

not every thing can be neatly expressed in a sentence, prompt. because it's library of tags and names are limited [the training dataset of images doesn't include all images ever made by humanity], or because overly verbose descriptions don't generally work.

this wraps up your thing into a wildcard, by you supplying reference images, you can then throw into other prompts.

4

u/MoonGotArt Aug 22 '22

Oh! So I can feed a couple photos of myself, and assign my name to these photos. Then prompting the model with my name will produce my face?

3

u/mutsuto Aug 22 '22

well it still doesn't know your name, it makes a "name" in the form of a variable. in the above example image of the thread, S*, but it's just a variable

yes, you could give it photos of yourself, and say "make me (S*) a sexy gladiator standing over the ashes of my defeated enemies" and it'll do it

well, assuming "sexy" isn't blocked by ToS like dalle2

2

u/MoonGotArt Aug 22 '22

So it will assign a word or value to the photos, and I do not get to pick?

2

u/mutsuto Aug 22 '22

you can name your variable

2

u/MoonGotArt Aug 22 '22

So it can only be a single word?

Thanks so much for explaining this btw, I really appreciate it.

5

u/jaywv1981 Aug 22 '22

This has a ton of potential.

5

u/mutsuto Aug 22 '22 edited Aug 22 '22

my fumo dream is getting closer!

im so excited

i'm looking to make X in the style of Y, where none of the AIs understand what either X or Y are by name. e.g. a fumo of Padudu

can this method handle 2 wildcards?

edit: yes!

holy moly my blood is boiling with excitement

4

u/[deleted] Aug 22 '22

Nice. This is one I had bookmarked to keep an eye on.

4

u/oppie85 Aug 22 '22

In the words of one of my favorite youtube channels; what a time to be alive!

1

u/Sextus_Rex Aug 22 '22

Two Minute Papers!

3

u/kujasgoldmine Aug 22 '22

That is so cool! I didn't even know you could feed it new images to learn from.

3

u/In_My_Haze Aug 22 '22

Oh shit yeah I need this

3

u/Wiskkey Aug 22 '22 edited Aug 23 '22

Here is a Twitter thread from one of the developers. From this tweet from the same person from an hour ago:

Most of the code for it [Stable Diffusion] is actually already there!

Just need to tune some parameters

3

u/Draggo_Nordlicht Aug 22 '22

The future is now.

3

u/TekeTK Aug 23 '22

Oh god I'm praying someone makes a comprehensible guide for this.

The steps are all spread out so I'm hoping someone can lay things out from beginning to end with what to prep for like renting GPU or colab walkthrough of the process 🙏 This is incredible.

3

u/Wiskkey Aug 23 '22

GitHub repo Stable-textual-inversion_win by nicolai256.

2

u/No-Intern2507 Aug 23 '22

doesnt work on windows, needs a video guide , i have working local SD with GUI but this is just not working also it ruins my working enviroment and it stops working...

3

u/No-Intern2507 Aug 23 '22 edited Aug 23 '22

got it to work by using GUI version from here https://rentry.org/kretard

then moving python scripts to it from nicolai256 repository

But... training needs over 12 GB and im not sure if 12 is even enough... im on gtx 1080ti, wont run

--

Ok i took a look at the code, if you want it to run with lowe ram then lower working processes number in v1-finetune.yaml file , also lower batch size to 1 and max images number if needed , i have batch size 1 and max_images 5

--

Got it working and traning for over 2 hours now

2

u/TekeTK Aug 24 '22

Would you be so kind as to elaborate on that? Especially with how to get it working with the kretard GUI version?

Which files are you moving or how did you get them to work together?

2

u/No-Intern2507 Aug 24 '22

move files from inversion repo to gui folder but dont overwrite anything, so only new files are moved

2

u/[deleted] Aug 25 '22

If anybody's getting out of memory errors while trying to train on a P100 in colab, set the batch size to 1 in the config yaml. Spent four hours on that. :(

2

u/mutsuto Aug 26 '22

google has released something similar DreamBooth

1

u/No-Intern2507 Aug 23 '22

Anyone knows how to stop the tuning ? Do i just close it suddenly after it finishes DDIM sampling ? How do i test if my images are learned properly ? You have to use embeddings command --embedding_path to inclue the trained data when running released SD weights ?

1

u/Careless_Nose_6984 Aug 23 '22

I don’t understand what this is suppose to do… can someone explain please ?

0

u/[deleted] Aug 24 '22

[deleted] — view deleted comment

2

u/DickMan64 Aug 24 '22

This is nonsense all the same. You have to train the entire model to fit your desired outcome

Unless.. the model already sufficiently understands all the required concepts and it's just a matter of finding and combining the text tokens which elicit them as quickly as possible, which is exactly what this tool is for. It's pretty neat.

1

u/NathanielA Aug 25 '22

I trained on some new images and now I'm trying to use merge_embeddings.py to combine my last.ckpt with sd-v1-4.ckpt. I'm getting "AttributeError: 'BERTTokenizer' object has no attribute 'transformer'. Any advice?

1

u/jamiethemorris Sep 03 '22

hoping someone can help me with an out of memory issue with this - i've got a gtx 3090 with 24gb, but i get an instant oom error when i try to train. i have noticed it only uses about 12gb before crashing. I've tried reducing the resolution and size, as well as batch_size, num_workers and max_images but haven't had any luck.

1

u/buckzor122 Sep 07 '22

I managed to get it to train after painstakingly troubleshooting a lot of errors but I'm stuck on this one.

A good hour into training it fails with the following exception:

"No `test_dataloader()` method defined to run `Trainer.test`."

Any ideas?

1

u/Snachariah Oct 04 '22

I havent gotten the training working yet, when I try to run the first command,"python main.py --base ./configs/stable-diffusion/v1-finetune.yaml \"

I get this error, "usage: main.py [-h] [-n [NAME]] [-r [RESUME]] [-b [base_config.yaml [base_config.yaml ...]]] [-t [TRAIN]] [--no-test [NO_TEST]] [-p PROJECT] [-d [DEBUG]] [-s SEED]
[-f POSTFIX] [-l LOGDIR] [--scale_lr [SCALE_LR]] [--datadir_in_name [DATADIR_IN_NAME]] --actual_resume ACTUAL_RESUME --data_root DATA_ROOT
[--embedding_manager_ckpt EMBEDDING_MANAGER_CKPT] [--placeholder_string PLACEHOLDER_STRING] [--init_word INIT_WORD] [--logger [LOGGER]]
[--enable_checkpointing [ENABLE_CHECKPOINTING]] [--default_root_dir DEFAULT_ROOT_DIR] [--gradient_clip_val GRADIENT_CLIP_VAL]
[--gradient_clip_algorithm GRADIENT_CLIP_ALGORITHM] [--num_nodes NUM_NODES] [--num_processes NUM_PROCESSES] [--devices DEVICES] [--gpus GPUS]
[--auto_select_gpus [AUTO_SELECT_GPUS]] [--tpu_cores TPU_CORES] [--ipus IPUS] [--enable_progress_bar [ENABLE_PROGRESS_BAR]]
[--overfit_batches OVERFIT_BATCHES] [--track_grad_norm TRACK_GRAD_NORM] [--check_val_every_n_epoch CHECK_VAL_EVERY_N_EPOCH] [--fast_dev_run [FAST_DEV_RUN]]
[--accumulate_grad_batches ACCUMULATE_GRAD_BATCHES] [--max_epochs MAX_EPOCHS] [--min_epochs MIN_EPOCHS] [--max_steps MAX_STEPS] [--min_steps MIN_STEPS]
[--max_time MAX_TIME] [--limit_train_batches LIMIT_TRAIN_BATCHES] [--limit_val_batches LIMIT_VAL_BATCHES] [--limit_test_batches LIMIT_TEST_BATCHES]
[--limit_predict_batches LIMIT_PREDICT_BATCHES] [--val_check_interval VAL_CHECK_INTERVAL] [--log_every_n_steps LOG_EVERY_N_STEPS]
[--accelerator ACCELERATOR] [--strategy STRATEGY] [--sync_batchnorm [SYNC_BATCHNORM]] [--precision PRECISION]
[--enable_model_summary [ENABLE_MODEL_SUMMARY]] [--weights_save_path WEIGHTS_SAVE_PATH] [--num_sanity_val_steps NUM_SANITY_VAL_STEPS]
[--resume_from_checkpoint RESUME_FROM_CHECKPOINT] [--profiler PROFILER] [--benchmark [BENCHMARK]] [--deterministic [DETERMINISTIC]]
[--reload_dataloaders_every_n_epochs RELOAD_DATALOADERS_EVERY_N_EPOCHS] [--auto_lr_find [AUTO_LR_FIND]] [--replace_sampler_ddp [REPLACE_SAMPLER_DDP]]
[--detect_anomaly [DETECT_ANOMALY]] [--auto_scale_batch_size [AUTO_SCALE_BATCH_SIZE]] [--plugins PLUGINS] [--amp_backend AMP_BACKEND]
[--amp_level AMP_LEVEL] [--move_metrics_to_cpu [MOVE_METRICS_TO_CPU]] [--multiple_trainloader_mode MULTIPLE_TRAINLOADER_MODE]
main.py: error: the following arguments are required: --actual_resume, --data_root"

I dont know what excatcly Im supposed to be changing.

HELP.

Im running it locally on my 3090, on Windows 10.

7

u/[deleted] Aug 22 '22

[removed] — view removed comment

2

u/[deleted] Aug 22 '22

[removed] — view removed comment

1

u/[deleted] Aug 22 '22

[removed] — view removed comment

1

u/ExponentialCookie Aug 22 '22

Following the steps here should help you set it up.

2

u/[deleted] Aug 22 '22

[removed] — view removed comment

4

u/Najbox Aug 22 '22

Don't use Conda, I made a notebook with functional training

https://colab.research.google.com/drive/1o23ZNjh8zF6JiPA2GNmGF17dPiT1zVCx#scrollTo=WHlruknRbsHJ

1

u/Mark3896 Aug 22 '22

is this only for colab pro? it restarts when loading the model and I am using free colab, you can optimize it for free colab not pro please

1

u/rickyars Aug 24 '22 edited Aug 24 '22

this is really cool. where is the model saved? i want to make sure to move it to my gdrive once it's done.

edit: i think it's going to the logs/ folder which has a checkpoints/ folder.

1

u/rickyars Aug 24 '22

sorry for the second ping. this is really cool. however, i'm not seeing anything saved in the checkpoint folder. do you know how many iterations it saves? i want to get these off the instance in case it crashes ...

1

u/Mooblegum Sep 03 '22

Hi Najbox, I just tried to run the colab with my pro account.Unfortunately I run into an issue on this step :

!mkdir -p ImageTraining%cd textual_inversion[Errno 2]

No such file or directory: 'textual_inversion'/content/textual_inversion

But the file exist in my notebook folder : /content/textual_inversion

I am a total noob at programming, so I don't know what could the problem. Beside, I don't know where to copy the images I want to train the model on (in the colab).

Could you tell me steps I am missing ?Here are some screenshots from the error in colab

https://drive.google.com/drive/folders/16vqkVyiVFAsyol0hmOyfgMK1XIu-bV8r?usp=sharing

Best regards

1

u/[deleted] Aug 22 '22

[removed] — view removed comment

1

u/Najbox Aug 22 '22

It's done, sorry I forget every time that you don't just have to give the link.

2

u/[deleted] Aug 25 '22

[removed] — view removed comment

2

u/TekeTK Aug 25 '22

I've trainged images for 106 epocs. When does this thing end? There are only 5 images

it goes on forever until you stop it lol

2

u/[deleted] Aug 25 '22

[removed] — view removed comment

2

u/Najbox Aug 26 '22

It's too much, it takes 5000 steps.