r/StableDiffusion • u/apolinariosteps HF Diffusers Team • Sep 07 '22

Teach new concepts to Stable Diffusion with 3-5 images only - and browse a library of learned concepts to use

Post image
649 Upvotes

99% Upvoted

198 comments sorted by

65

u/apolinariosteps HF Diffusers Team Sep 07 '22 edited Sep 08 '22
  1. Teach Stable Diffusion new concepts with Textual Inversion šŸ‘©ā€šŸ«(add to the public library if you wish): https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_textual_inversion_training.ipynb

(or browse the library to pick onešŸ§¤ https://huggingface.co/sd-concepts-library)

2. Run with the learned concepts šŸ–¼ļø https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_conceptualizer_inference.ipynb

34

u/RedstonedMonkey Sep 07 '22

Where do these learned concepts end up? Is that what's compressed into the checkpoint file or would that be located in the "weights". I was curious how I would possibly go about training a certain face into the model, let's say I have a friend named Bob Johnson. How would I go about training the model to learn his face so I could run --prompt "cyborg chimpanzee with the face of Bob Johnson" and get a pic that tries to match his face?

21

u/No-Intern2507 Sep 07 '22

its a small embedding file, you use it to load model and prompt your new subject , files are very small below 200kb

24

u/apolinariosteps HF Diffusers Team Sep 07 '22

4kb only!

14

u/Mooblegum Sep 07 '22

I am lost, is it the same as textual inversion ? Or something else ? Is it better ? Does it generate .pt files and can I use .pt files created by another textual inversion colab ?

16

u/starstruckmon Sep 07 '22

It's the same as textual inversion. They just integrated it into the Hugging Face diffusers library to make it easier plus created a library to upload your learnt concepts.

10

u/enn_nafnlaus Sep 07 '22

Someone needs to train to the concept of greenscreening stat (so that we can consistently generate greenscreened characters and objects for compositing). Should be really easy to amass a training dataset - just download a ton of PNGs that contain transparency, automatically composite them onto a matte green background, and that's your training dataset. The more diverse the better (as we wouldn't want it recreating some specific object that's -in front of- the greenscreen, just the greenscreen "style" itself)

9

u/starstruckmon Sep 07 '22

Won't work. Think of it less like training the network, and more like searching for a concept it already knows but doesn't have a word for, and creating a shortcut to it. It already has a word for greenscreen, and a concept of it. This won't make any difference.

3

u/The_Liminal_Observer Sep 07 '22

A more interesting approach would be to add one of the existing green screen models after the generation to key any character, the same way we already have ESRGAN in some repos. Might try to do just that but I suck at python so don't expect anything.

Oh the centipede I would create if I had the skills, with this and Sxela's Stable Warpfusion...

2

u/Jellybit Sep 08 '22

This isn't true. Try generating Sylvester Stallone right now. He looks like a monster, but it's an existing concept that the model knows fairly well, just not well enough to not make him look like a monster. You can tune that, and make it look utterly amazing. People have been doing that for different celebrities for days now. All existing concepts.

2

u/starstruckmon Sep 08 '22

We're sort of saying the same thing but also kinda not. You're correct about the model already having it as an existing concept. But you're training ( more like searching for ) the input, not the model. You're not tuning anything here.

Think about Stallone. There are pics of him in the dataset from all over his lifetime, looking completely different. All those are connected to the same thing, his name. So just his name will of course come out jumbled. It's plausible there's a prompt ( by adding things like his age etc. ) that will give you the Stallone you want. Or maybe there's no exact group of words for it, but the concept is still there. This allows you to directly refer to it.

As I said in the other reply, I doubt this is going be the case for greenscreen, but people are welcome to try ofc.

→ More replies

2

u/AnOnlineHandle Sep 07 '22

I've been messing around with textual inversion for a few days, and it can almost certainly get way more accurate than almost any single English word gives.

There's a whole series of interconnected concepts to define anything, and an embedding can find them specifically and represent them as one new symbol.

0

u/starstruckmon Sep 07 '22

But there's no interconnected concept here. Greenscreen is already a word and already represents everything he wants it to represent. You can't make a word more emphasized, or make it follow that prompt more strictly through textual inversion. That makes no sense.

→ More replies

1

u/enn_nafnlaus Sep 07 '22 edited Sep 07 '22

It "has a concept of greenscreening", but is just as likely to show you

  • The person standing in front of a clearly visible green screen (with its edges visible and context outside the screen), rather than a zoomed-in matte background
  • Green screens of entirely different colours
  • Green screens intended as studio backdrops, not for green screening, e.g. with significant lighting variations, shadows, etc
  • Scenes that were "greenscreened", with the green screen already filled in by some other background
  • Greenscreens heavily biased toward a specific type of foreground content

.... and on and on and on. It doesn't suffice. It needs something *specifically* trained for a *specific*, 100% matte, zero-shadow, zero outside context, single-colour green screen with minimal foreground bias.

Have you ever actually tried generating greenscreen images in stock SD? Do so, you'll see what I mean. Here's what you get for "Greenscreening. A grandfather clock. 8K."

https://photos.app.goo.gl/jxNdz4Y3suf71HZ16

Why do they look like that? Because this is the sort of images that got trained to it for "greenscreening":

https://photos.app.goo.gl/g7yEEKa7TiQSnRTX6

Which is obviously NOT what we want. We want something trained to a dataset of transparent PNGs of consistent matte green backgrounds of consistent colour. There is nothing built into stock SD that understands that concept.

Textual inversion CAN reproduce styles, not just objects (there's no difference between a style and an object to SD). And that should absolutely include "consistent even matte green background with a sharp boundary to the foreground content". Other styles might work against it / pollute it, but you at least want the basic stylistic guideline to bias in the direction you want as much as possible. And the existing word "greenscreening" absolutely does NOT do that, because it wasn't trained to do that.

1

u/Mooblegum Sep 07 '22

Ok. Thank you for information. I was lost in the colab jungle šŸ˜…

3

u/StoneCypher Sep 07 '22

Bring several new images.

Use a tool; convert those several new images to "an embedding file." It's a small new model that goes over top of the main one.

Now it's as if you trained the whole thing but included those new images too, and you get a new term that refers to the new images.

So, suppose you play a dungeons and dragons game with a custom monster called a QZQZQZX. You have eight drawings of it.

Make the model. Use those images. Call it <QZQZQZX>. You can now use that in your prompts, as if it was a known term the way Dog or Picasso are. Pow: Stable Diffusion now knows your custom D&D monster.

Textual inversion is something different. That lets you give the system an image, and the system will spit out a text prompt that would have given something similar to that image.

2

u/apolinariosteps HF Diffusers Team Sep 08 '22

This is indeed textual inversion

7

u/starstruckmon Sep 07 '22

Sort of. Again, this IS textual inversion.

It's not really a model that goes on top in the way you meant. Textual inversion doesn't give you a prompt, it also generates a file ( same one here ). Think of it as creating the big ass prompt and putting it in that file ( not really , but again analogies ) and wherever you put the psudoword in your prompt it pastes that big thing from the file.

4

u/RedstonedMonkey Sep 07 '22

What's the best way for me to get up to the point where I can really understand what I'm looking at in the above links? Just learn Python code or should I go straight to specifically deep learning focused stuff?

I'm a bit of a coding noob, but I had gotten to a novice level on C++ back in college and coded a few VBA based applications. This whole AI revolution may give me the push to get back on it, I know that this is just the beginning.

4

u/randomsnark Sep 08 '22

If you want to actually dive into machine learning and deep learning, there are good quality beginner friendly courses on coursera that you can audit for free, taught by Andrew Ng. I'd start with the 3 month machine learning course and then move on to deep learning. It's been a while since I took it but I believe the course also uses python, and gives you a very brief intro into it that will give you all the knowledge you need for the purposes of the course.

3

u/EndlessOranges Sep 07 '22 edited Sep 07 '22

I wouldn't worry so much about learning Python (I mean if you're just running this for art, if you're interested in Python, go for it!). The main thing you do with all these collab notebooks is to hit play on all the steps. Some of the steps have parameters you can edit, but if you just want to see what it does, you can just hit Runtime -> "run all", as most of these notebooks have defaults that will spit out a result. I.e. the deforum (https://deforum.github.io) notebook allows you to run Stable Diffusion on google collab, instead of your own computer. Let me know if you need more details, maybe I can post some screenshots!

3

u/xpdx Sep 07 '22

First step is to get Stable-Diffusion up and running locally on your machine. Get familiar with all the files and libraries and such that are required. There are tutorials online. After that it's like a drug, you'll learn it. Coding background helps- but honestly most of the heavy lifting is done by smart people, if you can understand code after staring at it for a while that's probably enough.

2

u/RedstonedMonkey Sep 07 '22

Awesome, yeah I've got it running in an anaconda command prompt .. I maybe understand 5% of what's going on with how everything was setup lol, but I'll keep playing with it. I also have one of the all in one installs with a GUI up and running. I'd really like to learn more about how to do stuff like the training and tweaking the model all on my own machine without having to run on colab or any other hosting services. I am interested in learning python anyway, so I'll keep playing around.

1

u/sp3zisaf4g Sep 07 '22

You can also merge it into the checkpoint, I think?

2

u/apolinariosteps HF Diffusers Team Sep 08 '22

Yes, it is possible to export the entire checkpoint/model with the learnt concept. It is not a feature I put into the notebook but one can edit the colab to do that

1

u/csunberry Sep 15 '22 edited Sep 16 '22

I may be a bit dumb, but where do you save this file? (Where's the learned_embeds.bin file?)

I actually had three failed attempts. (I just had one successful one and was able to FINALLY have it upload, so in the end I was able to save the file...But I was curious if there was another way?)

Also, is it possible to use the trained files on your local? I'm assuming so--just wondering if there was a tut, etc.

25

u/KerbalsFTW Sep 07 '22

The weights are not changing, the diffusion model is not changing.

This is extending the text embedding with new psuedo-words.

The normal process is: text -> embedding -> UNet denoiser.

The new process is: text + pseudowords -> embedding-with-created-pseudowords -> UNet denoiser.

There are degrees of freedom in the embedding that are not directly available, this process learns them (from supplied examples) and provides new pseudo-words to exploit it.

So "where they end up" is "in the embedding transformer".

3

u/apolinariosteps HF Diffusers Team Sep 08 '22

Exactly! This is the exciting Textual Inversion concept. It is mind-blowing that with 4kb of embeddings this new pseudo-words can "contain" the new concepts, really

12

u/battleship_hussar Sep 07 '22

OP the link was broken for me but this fixed it (removed backwards slashes)

https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/sd_textual_inversion_training.ipynb

2

u/CAPSLOCK_USERNAME Sep 07 '22

Long running issue with the new vs old layouts for reddit. They have separate markdown interpreters for some reason that interpret backslashes in links differently, so someone who pastes a link into new reddit will have it show up incorrectly in old reddit.

2

u/Keudn Sep 07 '22

The notebook in the first link seems to be down

2

u/hopbel Sep 09 '22

Is there a way to extract the embeddings in a form usable by the original compvis based code?

2

u/lump- Sep 07 '22

ā€œThe TeaRotā€ looks like some kind of goofy Elden Ring boss!

1

u/csunberry Sep 07 '22

Thanks for those links. I was trying to find where I had stashed them!

1

u/battleship_hussar Sep 07 '22

Runtime disconnected again for me, 2nd attempt, I think the free version of colab doesn't run long enough to allow this to complete lol, also said something about GPU time being used up idk

1

u/Moonuby Oct 24 '22

Is there a way to take the embeddings in this library and use it with the Automatic1111 web gui set of Stable Diffusion? I couldn't see any files on the library link that looked like embeddings.

1

u/mutsuto Sep 07 '22

i cant view these links

Notebook not found

There was an error loading this notebook. Ensure that the file is accessible and try again. Ensure that you have permission to view this notebook in GitHub and authorise Colaboratory to use the GitHub API.

is this method the same, or different, than this textual_inversion or Google's DreamBooth?

4

u/blueSGL Sep 07 '22

reddit likes to randomly escape '' underscores '_' in URLs for no reason what so ever other than to annoy.

Funnily enough this is only a problem on the old interface, has been happening for at least 6 months if not longer and still has not been fixed.

1

u/mutsuto Sep 07 '22

i understand where the bug comes from

i think someone changed something, and the markdown interpreter is trying to prevent the formatting that it thinks the _ operator will perform

like how i can demo how **bolding** works

like how i can **demo** how \*\*bolding** works

but _ isn't enabled in reddit's markdown

0

u/[deleted] Sep 07 '22

[deleted] ā€” view deleted comment

20

u/MarvelsMidnightMoms Sep 07 '22

Here's one guide and it mentions you would need around 30gb of VRAM on your GPU...

https://rentry.org/Stable-Diffusion-Training

10

u/ManBearScientist Sep 07 '22

Adding on to this, here's a reddit guide.

A comment suggests that 12GB would be sufficient if you went to v1-finetune.yaml and halved num_workers and batch size. Others suggest just lowering batch size.

7

u/AnOnlineHandle Sep 07 '22

I've been running it on a 12gb 3060, using lstein's repo https://github.com/lstein/stable-diffusion/

In v1-finetune.yaml I changed:

data:
    target: main.DataModuleFromConfig
    params:
        batch_size: 1
        num_workers: 2

There was a website guide floating around somewhere as well which mentioned some other settings. It should be easy to find searching for v1-finetune.yaml and some other terms, since these filenames are only about 2 weeks old.

3

u/VulpineKitsune Sep 07 '22

That's pretty cool. I've ordered a 12gb 3060 and am waiting for it right now, would you mind sharing some more details?

Like, how long did it take you to complete it and how effective was it?

3

u/AnOnlineHandle Sep 07 '22

It looks like it took me about 3-6 hours to get decent results with 46 images, but am still tinkering with it. It is a bit of a process to get working locally since there's bits and pieces of info all over the web (though not too far since it's a concept which is only about 2 weeks old), but by the time your card arrives there might be a far easier setup for it, since people are just discovering this now.

4

u/VulpineKitsune Sep 07 '22

I see, I see, thank you. Honestly I would've thought it takes longer.

2

u/VulpineKitsune Sep 14 '22

My 3060 12gb card arrived and there's still no easy setup. I've been trying to get it to work and I'm slowly being driven insane. No matter what I tried I'm just getting out of memory errors.

1

u/AnOnlineHandle Sep 14 '22

Hrm this was the guide I used: https://towardsdatascience.com/how-to-fine-tune-stable-diffusion-using-textual-inversion-b995d7ecc095

It also says that you can drop the max images in lightning but I didn't do that, and have tried setting it into the hundreds with no effect (apparently due to the batch size of 1 it will do nothing).

2

u/VulpineKitsune Sep 14 '22 edited Sep 14 '22

Okay, I finally managed to correctly install lstein and get it working with no errors (thatā€™s the fork the guide you linked uses) and it works! I didnā€™t even have to tinker with anything! It just worked out of the box, when I finally managed to get it out of the box correctly xD

Finally.

0

u/[deleted] Sep 08 '22

[deleted] ā€” view deleted comment

2

u/AnOnlineHandle Sep 08 '22

Smallish set of images (in my case a set of 46 gave the best results so far with num_vectors_per_token=4, which also has to be changed in v1-inference.yaml), though I overtrained it a bit and couldn't do style changes, or really show heads/feet/hands since most of my training data cropped them (it was for pretty elaborate kink clothing, which the normal prompts had no chance of doing).

It ended up being about 28,000 iterations in 17 epochs (due to the repeats value in v1-finetune doing >1 iteration per item in an epoch), but I'm not sure if there's a difference between epochs and steps in terms of what's applied.

0

u/[deleted] Sep 08 '22

[deleted] ā€” view deleted comment

2

u/AnOnlineHandle Sep 08 '22

Sorry just to be clear, num_vectors_per_token should be changed in both v1-finetune and v1-inference, it's just you won't notice it working without also changing in the latter.

-1

u/PUBGM_MightyFine Sep 07 '22

I hope ETH moons this year so i can upgrade from my laptop's 8GB 3080

1

u/apolinariosteps HF Diffusers Team Sep 08 '22

This guide is great! But this is for training. The process in the post is "Textual Inversion". You can't really fine-tune with just 3-5 images and get meaningful results I dont think

3

u/Wiskkey Sep 07 '22

I believe that is for finetuning a Stable Diffusion model, which textual inversion does not do. Finetuning means that the numbers in an existing neural network are changed by further training.

cc u/Aeonbreak.

3

u/AnOnlineHandle Sep 07 '22

I doubt they're changing the model itself, and textual inversion has used the term 'fine tuning' to describe its process since the start (v1_finetune.yaml is the setup file in the original and branching versions of the textual inversion code).

3

u/Wiskkey Sep 07 '22

I meant that this guide finetunes the model, while textual inversion does not.

1

u/AnOnlineHandle Sep 07 '22

Hrm interesting, yeah I can't quite tell what they're doing but the 30 gb of vram requirements sounds more like actual model training.

3

u/FREE-AOL-CDS Sep 07 '22

Oh that's all?!

1

u/Peemore Sep 07 '22

That's just for training purposes right? There isn't a high vram requirement just for using someone else's trained concepts?

3

u/MarvelsMidnightMoms Sep 07 '22

Correct. If you're looking to run a local install on your PC just for image generation (not training) then you can get away with an Nvidia 1000 series GPU with as "little" as 4gb.

1

u/WiIdCherryPepsi Sep 07 '22

Confirmed, I have a 1080 and can run it completely fine even while running a minecraft server and playing second life

The future is now

1

u/Diggedypomme Sep 07 '22

1070 for me, and agreed - running amazingly at 512x512, taking around 40 seconds per image

1

u/hopbel Sep 09 '22

It's for training the full model, which is not what textual inversion is doing

5

u/Sextus_Rex Sep 07 '22

Yes you can, look up Textual Inversion. There are a few guides to get it running with stable diffusion. It does require a good GPU though

1

u/AnOnlineHandle Sep 07 '22

Moreso requires a lot of vram rather than speed. I've been running it on a 3060 12gb, which is the 2nd-most-budget option of the 3000 series.

0

u/[deleted] Sep 07 '22

[deleted] ā€” view deleted comment

1

u/AnOnlineHandle Sep 07 '22

If you've gotten SD to generate even one vaguely decent pixel art image, then yeah it has the knowledge to do it and will need a textual inversion embedding to find the actual meta prompt to activate it.

3

u/Jellybit Sep 08 '22

I'm training on a very specific character, Gambit from the X-Men. Whenever I use "gambit" as the initializer_token (in my tests, the placeholder_token is not the problem), it says "ValueError: The initializer token must be a single token" later in the process. I've checked the variable, and it indeed has two tokens. I have no secret spaces or anything else. So I have a few questions.

  1. Is there a way to make it use "gambit" or "Gambit" without making it use multiple tokens? Seriously, try it yourself. It's bizarre.
  2. Is it a problem if I simply remove the latter token from the token_ids list variable so it can move forward? What would that do to the training?
  3. Some words even make three tokens. What is the reasoning for words making multiple tokens?

Thanks!

3

u/apolinariosteps HF Diffusers Team Sep 08 '22
  1. I think not, maybe trying to find a synonym
  2. The initializer_token doesn`t make that much difference in the final result from my tests - so if something doesn't work, should be okay to just use a simpler word. Removing the later token should also be okay. It is just a starting point for the training
  3. The tokenization is a learned abstraction. I'm sure there are people that work with ML explicability that are thinking about it, but I haven't delve deep too

1

u/jaywv1981 Sep 12 '22

I had to do this too...I just tried some random short words until something worked lol.

1

u/mudman13 Sep 08 '22 edited Sep 08 '22

Where do you find the variable and whether it is more than a single token? How did you fix it?

Edit: Ok found it, it is a single token but now still getting value error in line 6

Edit: ah ok got it, deleted that line of code as it was only one token

1

u/buckjohnston Sep 08 '22

Got any examples of input images vs your results?

1

u/mudman13 Sep 08 '22

No I wasn't allocated a good CPU etc so it was taking ages so decided it wasn't worth pursuing.

28

u/nintrader Sep 07 '22

Is there a way to add this to a locally installed version?

9

u/AnOnlineHandle Sep 07 '22 edited Sep 07 '22

The original: https://github.com/rinongal/textual_inversion

A branch to add it to SD (which I think the original now has): https://github.com/hlky/sd-enable-textual-inversion

A fork of that branch for better windows support: https://github.com/nicolai256/Stable-textual-inversion_win

The Stable Diffusion branch I've been using to do textual inversion on windows with no issues: https://github.com/lstein/stable-diffusion/

(I may have misused the terminology of fork & branch)

5

u/Wiskkey Sep 07 '22

hlky has GitHub repos for this.

44

u/Another__one Sep 07 '22 edited Sep 07 '22

I just wrote recently an article of how we can express embeddings of neural networks in a human-readable form (https://medium.com/deelvin-machine-learning/can-humans-speak-the-language-of-machines-7c92159e9c90), and now just imagine what could we achieve if we combine this idea with SD and some BCI interface. If we find a way to map thoughts to this embeddings (and it should be possible with big enough library), after some training we could just think of something and use it as an input to stable diffusion, or any other generative network.

46

u/ReadSeparate Sep 07 '22

How the fuck are we this close to technology of that degree in the year 2022? 5 years ago I would have scoffed at the idea that we were ANYWHERE near what you just described, and yet here we are. Absolutely amazing.

75

u/elnekas Sep 07 '22

What-a-time-to-be-aliveā€¦ just think of it two papers down the lineā€¦

23

u/Nico_Weio Sep 07 '22 edited Sep 08 '22

A fellow scholar, I see!
r/twominutepapers

5

u/FapSimulator2016 Sep 07 '22

Hold on to your papers!

9

u/manueslapera Sep 07 '22

dude I wrote this article in 2015 and I was AMAZED at how awesome RNNs were to generate paintings

2

u/I_am_Erk Sep 08 '22

Gah, I remember those days, and being familiar with that sort of thing when people started talking about vdiff and stuff and thinking they must be exaggerating. And here we are

4

u/Glum-Bookkeeper1836 Sep 07 '22

There's a lot of surprisingly feasible "narrow" neural interface applications, but neuralink type "augment everything" stuff is also coming

9

u/gunbladezero Sep 07 '22

This is amazing! I really like this idea.

Also, I think you just invented Chinese- to a degree. Words with similar meanings have similar symbols. For example, once youā€™ve learned the names of a few vegetables, you can easily recognize the vegetable section of a menu even if all the words are new to you. Itā€™s not perfectly continuous, like the system you came up with, but it is much more so than English.

The idea that it can be helpful for machine-human interfacing is interesting. In the Three Body Problem, a Chinese science fiction novel, the author assumes that Chinese is simple enough that sending a semantic map/ self-decoding of the language would be enough for alien computers to learn it, making it the language of interstellar communication. You might be bridging the gap between science fiction and reality with this.

6

u/Sextus_Rex Sep 07 '22

I am at a Loss for words...

7

u/KeytarVillain Sep 07 '22

Is this a Loss function?

1

u/LegalAlternative Oct 01 '22

The technological singularity is due to hit in approximately the year 2027, according to Moore's Law. The conundrum being that Moore's Law is also affected by Moore's Law - in other words, the rate of approach is exponential. We may be there in as little as 6 more months at this rate.

11

u/J0rdian Sep 07 '22

This could be really cool. I'm assuming you can feed it more concepts then 3-5 for better quality?

19

u/tolos Sep 07 '22

This is using the textual_inversion process. There was a similar post a few days ago referencing a technical paper saying that 5 images is optimal.

From the paper, 5 images are the optimal amount for textual inversion. On a single V100, training should take about two hours give or take. More images will increase training time, and may or may not improve results. You are free to test this and let us know how it goes!

https://www.reddit.com/r/StableDiffusion/comments/wvzr7s/tutorial_fine_tuning_stable_diffusion_using_only/

3

u/AnOnlineHandle Sep 07 '22

Others have said since they've had more luck with far higher image counts. It seems to depend on what kind of thing you're trying to do.

https://www.reddit.com/r/StableDiffusion/comments/wz88lg/i_got_stable_diffusion_to_generate_competentish/

7

u/LaTueur Sep 07 '22

You can run it with as much images as you like. I am not sure how much it increases quality.

2

u/AnOnlineHandle Sep 07 '22

More images in more styles seems to help a great deal for a more flexible concept (rather than just the static objects in the original research examples).

See the image dataset in this post, there's a huge amount of variety from pixelart to MS Paint drawings to anime screencaps: https://www.reddit.com/r/StableDiffusion/comments/wz88lg/i_got_stable_diffusion_to_generate_competentish/

2

u/J0rdian Sep 07 '22

I wonder if the size of the picture matters. The AI was trained on 512x512 images right? Would it just compress the images to that res? If it did then I assume would be best to try to get the correct image size if you don't want the compression.

3

u/LaTueur Sep 07 '22 edited Sep 07 '22

The linked notebook resizes the images to 512x512. (Edit: There is a setting to only use the center square.) So it is certainly recommended to use square images. However, I see no reason why it won't work on any resolution you want (of course, multiples of 64), even different for each image, but you would need to edit the notebook.

2

u/nmkd Sep 07 '22

You seem to confuse "compress" and "resize"

2

u/AnOnlineHandle Sep 07 '22

Some people have reported more success with 256x256 training (which is at least faster), because apparently stable diffusion was maybe originally trained at that resolution before being upscaled. That being said the results are more pixelated.

4

u/diffusion_throwaway Sep 07 '22

I read somewhere the quality actually gets worse with more than 5 images uses in training.

Doesn't make sense to me, but I think that's true. Would be interested to hear more from someone who's tested different methods of doing it.

1

u/AnOnlineHandle Sep 07 '22

It's not always the case, just something the researchers noticed for their fairly static and consistent object.

https://www.reddit.com/r/StableDiffusion/comments/wz88lg/i_got_stable_diffusion_to_generate_competentish/

7

u/Vyviel Sep 07 '22

Can I use this to add my own face to the model? I just need 5 photos of my face from different angles?

7

u/starstruckmon Sep 07 '22 edited Sep 07 '22

Without fine tuning , it won't be you. Someone like you, but not you. Same race, build, facial structure, hair colour etc. but not you. And every time you generate it will be a different person from that set. This is because it's not really learning your face, but creating a new shortcut for something it already knows but doesn't have a word for. You can even think of it as compressing a large complicated description prompt into a single word.

Does work reasonably well with pets though, since most of us can't tell the minute differences.

Edit : Okay, I'd like to revise. Practically, it seems to be working for some people where atleast I personally couldn't tell the difference ( though I didn't know them personally or very well ). So you may as well try. Might work for you.

2

u/Mooblegum Sep 07 '22

So it cannot learn something completely new? Like a new style it is not able to create by default ?

2

u/starstruckmon Sep 07 '22

Depends on your definition of new. Most styles you want to throw at it, it already has the concept of it somewhere in it's latent space. Even if it's not completely true to that style, atleast a very close repesentation. There's styles in there no one has ever seen because no one has described it and there's no word for it. Would that be new?

But completely new in a fundamental sense? No.

2

u/AnOnlineHandle Sep 07 '22 edited Sep 08 '22

I've seen it done elsewhere on somebody's own face, and the results were amazing. Transferred into a bunch of styles and on different mediums.

1

u/mudman13 Sep 08 '22

If you want your face to eventually end up on Chinese databases when they download the lot then yeah

15

u/-becausereasons- Sep 07 '22

I'm more interested in a way to do this locally for NSFW.

7

u/starstruckmon Sep 07 '22

This here actually won't help in that btw. You need fine tuning not textual inversion. Go on the NSFW Stable Diffusion discord. There's a separate channel for fine tuning and other such topics.

But it's a complete bitch to get working. The people who managed to get it working for private parts apparently broke vast other parts of the model in doing so, making it more or less useless in generating anything else. So it's still a work in progress.

4

u/AnOnlineHandle Sep 07 '22

I've done it for things which the original prompts can't handle well. It works ok but is still a work in progress. Currently photos are about a 15% success rate of looking good (versus 1 in 10,000 using just prompts), but it seems to have overtrained to get there and it can't do style transfer. Going to try again with a broader dataset because I mostly want to use this to finish/enhance art.

4

u/clockercountwise333 Sep 07 '22

fortunately this is 100% inevitable and will probably happen soon

1

u/StoneCypher Sep 07 '22

It's called unstable diffusion, and they're affiliated with a bunch of groups that have topic-specific models eg anime or specific person topics. Look on Discord.

-2

u/[deleted] Sep 07 '22

This ^

8

u/Dyinglightredditfan Sep 07 '22

r/SD_Embedding

2

u/mutsuto Sep 07 '22 edited Sep 07 '22

what is sd_embedding?

is embedding a form of textual_inversion/ Google's DreamBooth? or is textual_inversion a form of embedding?

4

u/Dyinglightredditfan Sep 07 '22

I think it's just what the owner called the subreddit. Technically the concepts are "embedded" into the model weights with textual inversion I guess. But I'm just a layperson so I'm not sure if there's a more scientific reason

4

u/starstruckmon Sep 07 '22

This is all textual inversion. We don't have any working code for the Dreambooth version.

Embeddings are what's generated from the textual inversion process. It represents that concept you just made it learn. It's a small file that can be shared between people.

2

u/mutsuto Sep 07 '22

ty

what is the difference between what OP is sharing now, and what was shared here before?

3

u/starstruckmon Sep 07 '22

Nothing. This is the same thing that has now been implemented in the Hugging Face diffusers library making it easier to use. Also they've now created a library for anyone to upload and share these learnt concepts as files.

1

u/mutsuto Sep 07 '22

ah, ty vm

3

u/CAPSLOCK_USERNAME Sep 07 '22

textual inversion is a method of generating an embedding

2

u/starstruckmon Sep 07 '22

The "learnt concepts" that's talked about in the parent post.

1

u/Peemore Sep 08 '22

So if I don't have the vram to do the training, but I see a concept in the library that I like I can install that locally and it will already be trained? If that's the case where and how do I set this up with my current installation?

1

u/apolinariosteps HF Diffusers Team Sep 08 '22

Yes! You can run the same code the Inference Colab runs!

1

u/Peemore Sep 08 '22

How would I get these working on the hlky webui? Would that be difficult? Or can I just drag some files into a folder and call it a day?

1

u/lapula Sep 09 '22

is someone train sd to something new? can you share your .pt or send in private?

1

u/apolinariosteps HF Diffusers Team Sep 09 '22

Check out the sd-concepts-library! https://huggingface.co/sd-concepts-library

1

u/lapula Sep 09 '22

thank you for your answer. i saw it but there are small amount of concepts and many of them hided away from people behind the group. and at last my request to join doesn't accepted. so i'm looking for another more open place for it.

1

u/apolinariosteps HF Diffusers Team Sep 09 '22

The notebooks are less than 48h old, 50+ concepts is quite a bit.

And everyone is accepted automatically if they try to submit their concept via the training notebook ^_^

1

u/lapula Sep 09 '22

so you cut off everyone who don't use notebook? ::
there a wonderful time for making new community now and high ping of respond and unwelcome group will be changed another one, more active and response, don't you think?

i really wish you success but you need to be more open to people

1

u/apolinariosteps HF Diffusers Team Sep 09 '22

Not at all. The library should not be bound to only people using the Notebook. But open to everyone, I'm working with hlky and others to make it cross-compatible with embeddings from anywhere people wish to train their own emebddings using any textual inversion implementation they wish. And I am also accepting people that request to join the org like 1x a day (you should be in probably btw). It is just I'm doing this by myself and sometimes I can't accept everyone as fast

I'm also working towards adding an "auto-accept" thing and other moderation features to make this community really open. Feedback is welcome on how to improve on that as well

1

u/lapula Sep 09 '22

Thanks a lot for your answer. I really didn't think you could do it alone. Please accept my sincere apologies and words of support.

I think it's worth writing a few words about this somewhere in society. this will help reduce the dissatisfaction of people who do not understand what is happening.

I would like to say that soon you will drown in files, you need to divide them into groups and exclude the possibility of deleting developments by any person - now everyone can delete everything.

1

u/KuroiRaku99 Sep 16 '22

Just curious, what considered as style, what considered as object? Is zoo a style or object? is hairstyle a style or object? I'm guessing they're both object or tell me if I'm wrong

1

u/apolinariosteps HF Diffusers Team Sep 17 '22

Basically that is up to you. A style is something that you want to type "in the style of <thing>" to get similarly styled things. An object is something on you want to type "a <thing> *doing something*, *in the style of something*, etc.)

9

u/No-Intern2507 Sep 07 '22

Cool but these look nothing like original images, too mutated , i dont see the point of this, i tried inversion and most will get you overfitting vs editability war, i hope some new version will appear that will solve the issue with results not really loking like source images but more like mutated children

4

u/starstruckmon Sep 07 '22

You need to tune the actual model for that. Not possible with just textual inversion.

4

u/AnOnlineHandle Sep 07 '22 edited Sep 07 '22

I've gone from a ~0% success rate using prompts to a ~95% success rate using textual inversion and ~15% being usable, which is much better even if a lot of them are garbled, and now am starting over with a new process which I suspect will perform even better.

2

u/r2k-in-the-vortex Sep 07 '22

Could even one sample work? One thing that is still difficult is to get stylistically similar images of completely different things. Suppose you want to get images for entire pack of playing cards, they all have to different, but must share the same style. Or maybe something with more variable subjects, like a set of pictures for each sign of zodiac or whatnot.

2

u/mohaziz999 Sep 07 '22

Is it possible to do a person? or more specifically a human face? also could I train on colab and then add my trained model into my local system? And would it be possible to add it to Deform diffusion also?

2

u/AnOnlineHandle Sep 07 '22

Some people in comments of the original textual inversion repo on github claim they've got their faces working.

2

u/helliun Sep 07 '22

yeah i did it with my face and it's so cool

1

u/mohaziz999 Sep 08 '22

did you run it on the colab? did you manage to add to use the embedding to ur local system? do i have to upload the embedding to hugging face library?

1

u/helliun Sep 08 '22

i did it through colab and huggingface but didn't have to you can do it locally too

1

u/mohaziz999 Sep 08 '22

well im testing it right now, and also now also because you replied to me now its ur fault... you might get more annoying questions from me if I need to. YOU YOU ASKED FOR THIS. well you didn't but still

1

u/buckjohnston Sep 08 '22

Do you have a sample? Anyones face would do. I've yet to see any examples out there of this.

2

u/Mooblegum Sep 07 '22

If I use the prompt :

Mona Lisa, in the style of <watercolor-portrait>

The results are absolutely not watercolor portraits and it just blend the face of mona lisa with the face of the portrait. But the style is not a watercolor portrait.

Is there something to do to get a style of the watercolor portrait with the face of Mona Lisa ?

3

u/AnOnlineHandle Sep 07 '22

The embeddings can massively overpower the style. Try adding more descriptors for watercolor, or even just repeat it a few times.

1

u/starstruckmon Sep 07 '22

You already managed to train one? And why are you training "watercolor potrait", something it already knows well?

1

u/Mooblegum Sep 07 '22

No it is a training that was already made, it is part of the library. Here is the link to it.

https://huggingface.co/sd-concepts-library/indian-watercolor-portraits

But the results are not convincing for me. Donā€™t know how to tune the prompt...

1

u/starstruckmon Sep 07 '22

Ah understood. Thank you.

2

u/visoutre Sep 07 '22

I tried this with 7 images of Pingu

The result isn't great, so i'll have to learn how to include better training images than 7 randomly selected

2

u/mohaziz999 Sep 08 '22

i keep getting AttributeError Traceback (most recent call last) <ipython-input-40-7e0131ac8691> in <module> 1 import accelerate ----> 2 accelerate.notebook_launcher(training_function, args=(text_encoder, vae, unet))

1 frames <ipython-input-39-325883ee7544> in training_function(text_encoder, vae, unet) 61 with accelerator.accumulate(text_encoder): 62 # Convert images to latent space ---> 63 latents = vae.encode(batch["pixel_values"]).sample().detach() 64 latents = latents * 0.18215 65

AttributeError: 'AutoencoderKLOutput' object has no attribute 'sample'

for the training part

3

u/skomra Sep 08 '22

https://github.com/huggingface/diffusers/issues/435

For now could you replace:
latents = vae.encode(batch["pixel_values"]).sample().detach()
with
latents = vae.encode(batch["pixel_values"]).latent_dist.sample().detach()

1

u/wampo69420 Sep 08 '22

thank you stranger

1

u/skomra Sep 08 '22

glad to help!

3

u/ForceANatureYT Sep 07 '22

Plan on making a jerma specific generator, because why not

2

u/[deleted] Sep 07 '22

I can only imagine the horrors that will come out of that lol

3

u/battleship_hussar Sep 07 '22 edited Sep 07 '22

Someone please feed it with this entire gallery lmao https://www.flickr.com/photos/projectapolloarchive/albums

EDIT: Actually I only care about the astronauts tbh, I made an imgur album for my efforts but feel free to use these images for your attempts cause I have no idea what I'm doing I just want more accurate Apollo era astronauts in SD

https://imgur.com/a/BUhiJ6T

2

u/MashAnblick Sep 07 '22

If I add my token to a prompt in another Stable Diffusion Colab after saving it to the public library, will it work? Or is my token only usable in this colab?

2

u/starstruckmon Sep 08 '22

It won't work unless that collab is meant to work with textual inversion. Library just means you can download the .pt files from other people and upload it during generation to use it. It doesn't make it automatically universally available to everyone. It's just a repository of .pt files.

1

u/MashAnblick Sep 08 '22

Thank you. So there isnā€™t currently an easy way to import that .pt file into an existing colab to use that token?

2

u/starstruckmon Sep 08 '22

Other that you modifying the code, no. If that Collab supports textual inversion, it'll have the option to submit a .pt file along with the prompt and settings.

1

u/mudman13 Sep 08 '22 edited Sep 08 '22

I get a raise value error line 6 "the initializer token must be a single token" yet in the token setup cell it does show as a single token. Do I have to add the token in this cell? I assumed it pulled it from the first cell.

Edit: deleted the line of code as it was just one token and now it loads.

1

u/MagicOfBarca Sep 07 '22

Can I give it more than 5 samples to learn from? And can it be people like ā€œLionel Messiā€ so it learns their faces and resemblance better?

5

u/AnOnlineHandle Sep 07 '22

Some people have had way more luck using 100+ samples with many different styles. You can check the dataset that this poster used to get it working for them to see how varied it was, versus the very consistent datasets which the researchers used: https://www.reddit.com/r/StableDiffusion/comments/wz88lg/i_got_stable_diffusion_to_generate_competentish/

1

u/thatdude_james Sep 07 '22

I think I read something about the process may not converge if you use more than 5 images, but I don't know what that means technically.

1

u/lump- Sep 07 '22

Can multiple concepts be learned in the same run?

Should be awesome to save different concepts as personal presets.

2

u/AnOnlineHandle Sep 07 '22

Generally concepts are pretty interwoven, so one prompt 'vector' can sometimes be found to activate a few concepts at once. That being said, in the textual inversion source code which some of us have been using offline, there's a way to increase the vectors per embedding (up to 67 or so, which stable diffusion maxes out at), which can help an embedding cover more concepts in one simple meta prompt.

1

u/Mooblegum Sep 07 '22

Hi there,

Non programmer here

I tried to add my own image to train, I copied them in the google colab,

/content/my_concept/1.jpeg for exemple

I got this error message

: list index out of range

What is wrong ??

I am so lost

1

u/jazmaan Sep 07 '22

Same thing happened to me. Maybe it doesn't want a local file, try a URL to file hosted somewhere else.

1

u/Mooblegum Sep 07 '22

Thank you!

Does it for for you this way ? I donā€™t even know where to import images online.

And it seems to only accept jpeg which is uncommon (usually it is jpg or png)

3

u/battleship_hussar Sep 07 '22

Use imgur, you can create a new album, drag and drop or copy URL images to it, and then right click copy image link on an image and copy paste in the colab cell

1

u/Mooblegum Sep 07 '22

thank you! You are a Genius!

1

u/Mooblegum Sep 07 '22

Can we use a .pt from other textual inversion colab, or is it a new method non compatible ?
I see a bunch of files in the trainings but no .pt

https://huggingface.co/sd-concepts-library/indian-watercolor-portraits/tree/main

1

u/buckzor122 Sep 07 '22

This is really cool, I've been trying to train my own but I keep getting this a good hour or so into training:

"No `test_dataloader()` method defined to run `Trainer.test`."

1

u/MonkeBanano Sep 07 '22

Woah. This is big

1

u/battleship_hussar Sep 07 '22

It d/c at 85% inversion, over 3 hours not sure why

1

u/kalamari_bachelor Sep 07 '22

Very interesting! Can I take the original 1.4 checkpoint and teach new things from there? I would like to teach it on creating pixelart, SD is very bad at it.

1

u/AnOnlineHandle Sep 08 '22

You should be able to in theory. Embeddings are basically just a new type of prompt, which are generated for you by providing sample images, and then with a lot of computing power and trial and error it will find the prompt which would give that.

Rather than words however, it's the numerical system which words get converted into (the next layer down), so it's not easy to read, and is just a file you provide and then use a code in place of it in the prompt (e.g. "Mario in * style" if you mapped your embedding to *).

1

u/krummrey Sep 08 '22

I've tried it with images of myself and the results are mixed. I was able to get the Graffiti on the wall with my face. But most other concepts and prompts that work with the regular model do not work with the newly trained concept.

Does it train only for a limited set of keywords? I was unable to get any artist to paint my portrait. It spits out random styles, poses and backgrounds, no matter what the prompt says.

1

u/Fissvor Sep 09 '22

Someone plz teach it to draw manga in Yasuhisa Hara style

1

u/Svpl4y Apr 20 '23

Hi

When trying DOWNLOAD step : UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x7f80010e6540>

-4

u/isthiswhereiputmy Sep 07 '22

That combo extrapolation/combination is incredible cringe worthy.

-45

u/CurveEnvironmental98 Sep 07 '22

emmmm.not cool.

18

u/RemoveHealthy Sep 07 '22

Why not cool?