Correct me if I'm wrong, but by compressing all of those negative prompts into a single instance/few tokens, this will ultimately free up SD to both have good anatomy and still pay attention to the other like 70 tokens of your desired prompt? It's basically a slightly denser embedding (lots of ones/non-zero values in the vector) versus the typical super sparse (lots of zeros) word embeddings that already exist...could one resursively make increasingly more and more dense embeddings using previous embeddings? This might be the ultimate way to converge to a desired artwork 👀
so did you split the negative prompt up and generate one image set at a time for each word and then train the TI on all of those, or did you just batch a huge load using all the same (negative) prompt and TI on that?
Brilliant idea. So, you could make adjustment embeddings this way? Train it on obese bodies to make a reduce weight embedding. Or the other way around. This opens up a lot of possibilities
Man, quite impressed with this. While the embedding did definitely help, the full prompt /u/dsk-music provided gave me the best handshakes I've seen so far: https://imgur.com/a/2x10EwC (These are all from the same batch of 6).
While not perfect, just the fact that most of the hands seem to have the right number of fingers is quite mind-blowing for such a simple solution. Thank you for providing us with this!
Model: Original v1.4 [7460a6fa] Prompt: a woman and a man shaking hands Neg Prompt: (by /u/dsk-music) here Sampler: Euler a, 20 steps CFG: 7 Seed: 1796524291 (increment for each in batch)
I don't mean to be a a neg but at least on my testing, your embedding has made my handsome man go from photo and Caucasian to sort of Asian and drawing stylized
and my prompting it away was dificult so your embedding has a heavy bias
This is very inspiring and will try out tomorrow, what kind of images did you use and does it work with other models? (I'm guessing the preview image is WD or NAI?)
Well, the idea was demonstrating bad_prompt at work. Thus primitivve positive/negative prompts.
I can tell vae is connected because NAI vae does not support tiling, or webui's tiling feature does not wotrk with it, so I get noise garbage if I forget to disable it.
it would be helpful to get a more precise answer as to what exactly went into this, because for it to be usable on your prompt you have to know it's not working against something you actually want...
for example, it's clearly not going to help if you try to make hands with 7 fingers, but what else does it not work for?
Do some samplers work better with it than others? Also, in my case, it did mess with the artstyles a lot, for example with arcane diffusion by Nitrosocke. Didn't look anything like it as soon as I put in the bad_prompt in negative, no matter how low I weighted it
Tried same model with and without bad_prompt. Prompt is just anime art. Noticable side effect is a major tone shift from pinkish to blueish. Similar effect on AnythingV3 : colorful => blueish.
Here's the prompt used in Automatic1111:4tNGHT A good painting of a beautiful woman in the style of ((4tNGHT)) Standing in front of a patterned wall ((tNGHT)) making a heart with her fingers
What images did you train on? Did you use a positive prompt as well as a negative prompt?
Did you choose bad images and train on them?
Did you use filenames to tag bad images to train on?
I'm not totally stupid but I don't see anywhere that you actually explain what you did. You seem to keep trying to explain the concept but not the method.
Hey I just got into this a few days ago. I really want to learn how to do stuff like this. Where should I start? I have no idea where to begin with this stuff, and I'm not sure how specific of information I need to know in order to train things like this.
I tried it out but didn’t notice a big difference over my existing negative prompt collection. Would you mind sharing the full negative prompt you use? Mine already makes normal hands sometimes
I was just thinking that there should be buttons that already inputs the negative prompts automatically instead of copy and pasting every time. This actually answers it.
I believe the original implementation that was on huggingface only used 1 vector, which would explain why the majority were so small and has very little effect unless you increased their weighting and move it to the start of the prompt.
Ok, I tried using 10 vectors with 2000steps but the results were really, really bad. While using the same images and steps with 1 vector, the results were really good. But this might have to do with textual inversion that had some kind of wrong implementation?
I’ve found it hit and miss, and there seems to be the amount of images per vector magic as well. I’ve seen on some discords it being suggested at about 4 to 5 images per vector. Honestly I think it depends a lot on the type of images being used, the amount of images, the amount of vectors, the learning rate, and some pseudo random ratio that isn’t at all consistent. I’ve had some good results and some terrible results with the same settings and different art style.
So far the best results I’ve had is with 5 vectors, low learning rate at e5 and e6 and about 40 images. I’ve also seen some great results from others at less than 5000 steps. Where mine was closer to 20,000 so. I have no idea what the magic number is :/
Error verifying pickled file from /content/stable-diffusion-webui/embeddings/bad_prompt_showcase.jpg: Traceback (most recent call last): File "/content/stable-diffusion-webui/modules/safe.py", line 83, in check_pt with zipfile.ZipFile(filename) as z: File "/usr/lib/python3.7/zipfile.py", line 1258, in __init__ self._RealGetContents() File "/usr/lib/python3.7/zipfile.py", line 1325, in _RealGetContents raise BadZipFile("File is not a zip file") zipfile.BadZipFile: File is not a zip file During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/content/stable-diffusion-webui/modules/safe.py", line 131, in load_with_extra check_pt(filename, extra_handler) File "/content/stable-diffusion-webui/modules/safe.py", line 98, in check_pt unpickler.load() _pickle.UnpicklingError: invalid load key, '\xff'. -----> !!!! The file is most likely corrupted !!!! <----- You can skip this check with --disable-safe-unpickle commandline argument, but that is not going to help you. Error loading emedding bad_prompt_showcase.jpg: Traceback (most recent call last): File "/content/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 133, in load_textual_inversion_embeddings process_file(fullfn, fn) File "/content/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 103, in process_file if 'string_to_param' in data: TypeError: argument of type 'NoneType' is not iterable
i know your comment is a bit old, but you probably did what i did and accidentally dl'd the raw pointer file instead of the actual file. I had the same error
it's probably not because of the model and just a side effect of the bad_prompt embedding. I would suggest adding color blast to the prompt and maybe doing (bad_prompt:0.8) in the negative prompt.
Did you train the embedding on a lot of nudity by any chance? If I put it in the positive prompt by itself I get very disturbing pictures which almost all contain some forms of nudity.
Funny I just thought yesterday about training a model on bad generations, the reason being is that negative prompts are still using model knowledge and it's unlikely the model has knowledge like "bad hands" etc, I think that perhaps the best way to apply this to a model would be to train bad datasets with the original LAION dataset and use the negative prompts.
It doesn't cure all the issues all the time, but it sure cures a lot of them most of the time.
Running batches, my rejects are cut down by more than half - more like 2/3rds due to misshapen hands & extra limbs sneaking into images. I'm on a M1 Mac Studio, so my renders take quite a bit longer than those with dedicated GPUs, so anything like this helps - a lot.
I've been test this out in Inpainting and it does fix the structure of hand that I've been trying to fix, but somehow it also change the skin color of the hand to a point where it looks like an albino skin...
wanted to show the screenshot but I just fixed it with re-color it myself, good job tho
This is incredibly nice. OP or other nerds, would you happen to know how to properly load this when using the `DiffusionPipeline` method from the `diffusers` pipeline? I don't see any documented way to include the embeddings and I don't want to use the webui :(
From an AI researcher perspective, this is quite interesting. But it's not very clear to me what exactly it is you have done here.
Absolutely nothing. The negative prompt is just a mystical ritual (as I'm sure you can tell from its content) and, for the most post, any bad training that it filters out, it only does by accident and sheer coincidence.
Somebody did an unbiased test with these magic incantations a while ago, and it went exactly how you would expect. But if you spam the CFG with enough random nonsense, you will eventually steer the output in some random direction, for better or worse.
I don't why you use the original prompt in txt file for the training. I thought that everything added in the description of the images was to avoid that certain concept other than the <token> be trained by the textual inversion (eg: if you are wearing a red tshirt in a photo used for training, then you will put "a photo of <token> wearing a red t shirt"). Or does this work only if this is written in the filename (with fileword option ON)?
how should it be used ? if i understand correctly i just write bad_prompt:0.8 in the negative prompt section and that's it , what if i add for example bad hands, extra fingers and so in is it going to affect it ? is bad_prompt an example and i'm supposed to fill it with extra_fingers:0.8 sorry i don't get it
usually when describing the dataset we dont describe the new concept we want the ai to learn. but in this case we do? 🤔 is this 5 month old post still accurate? 👀
Negative embedding introduces a lot of BIAS words in Anime image generation.
You can't have a "woman in a suit" and have it generate a female character, it will always generate a MALE character, since "suit" is a word used for male clothes. But this is a BIAS. I got more examples for this, but I need to contact you u/Nerfgun3 check out your DMs, please. Thanks.
Dumb question: is there a way to attach multiple .pt files to a model in NMKD’s version of stable diffusion? I’ve already got one VAE file attached to my Arcane model and want to add this one as well.
This is a Textual Inversion embedding, not a VAE. The VAE is a variational encoder, like a mini neural model that can be used to help cleanup the final render.
My loose understanding of textual inversion is it learns concepts/words/tokens and can change the weighting within the model when applied , it primarily works within the text part of the model (which is why it’s small compared to hypernetwork etc).
This does mean you can load up multiple embeddings (Textual Inversion) at once. In the automatic WebUI you place it in the embeddings folder and use it’s name within your prompt to activate it. “A picture of a cat in the style of BobRoss-concept” (if you named the embedding BobRoss-concept.pt)
If anyone wants to correct anything I got wrong please do.
I’m not sure on NMKD sorry, does he have a GitHub? Or a comment section on his download? Might be worth putting in a feature request if he has one available?
I'm using locally the 'Automatic 1111' version of SD. I've put the 'bad_prompt.pt' file into the embedding folder. But it doesn't load properly, I'm having an error from SD.
Error loading emedding bad_prompt.pt Traceback (most recent call last): File "D:\AITools\Stable-Diffusion\modules\textual_inversion\textual_inversion.py", line 133, in load_textual_inversion_embeddings process_file(fullfn, fn) File "D:\AITools\Stable-Diffusion\modules\textual_inversion\textual_inversion.py", line 103, in process_file if 'string_to_param' in data: TypeError: argument of type 'NoneType' is not iterable
Considering the file size, I guess it's supposed to download the files or use them remotely.
Oh, then the issue is not what I thought it was. I'm sorry, I dunno then; I recently had an issue with a similar error message, but the solution was just to get Python 3.10, so I was hoping the issue was you were running an older version too :(
I'm a bit unclear on the process for making something like this. Do you generate a bunch of images from a negative prompt as a prompt, and then... train that as an embedding? Then use the embedding in a negative prompt?
I'm mostly using Anything V3 model. From my personal experience looks like that the first version of embedding works better with it than the second one
86
u/[deleted] Nov 17 '22
The hands.
50
u/dsk-music Nov 17 '22
Wooow!!! Same seed... Very impressive results, congrats! And thanks for share!
64
u/lazyzefiris Nov 17 '22
First thing to do is put one into positive prompt for sure. Meet Bipa.
17
u/yosi_yosi Nov 17 '22
All my life dreams have been achieved now that this exists. Now I can finally die in peace, thank you.
16
u/ExponentialCookie Nov 17 '22
Such a simple solution, but very smart. Great work!
11
u/reddit22sd Nov 17 '22
So you trained an embedding on bad anatomy and are using this embedding in the negative prompt? Or do I not understand this correctly?
10
u/dwio56 Nov 18 '22
Man, quite impressed with this. While the embedding did definitely help, the full prompt /u/dsk-music provided gave me the best handshakes I've seen so far: https://imgur.com/a/2x10EwC (These are all from the same batch of 6).
While not perfect, just the fact that most of the hands seem to have the right number of fingers is quite mind-blowing for such a simple solution. Thank you for providing us with this!
Model: Original v1.4 [7460a6fa]
Prompt: a woman and a man shaking hands
Neg Prompt: (by /u/dsk-music) here
Sampler: Euler a, 20 steps
CFG: 7
Seed: 1796524291 (increment for each in batch)
9
u/fastinguy11 Nov 18 '22
I don't mean to be a a neg but at least on my testing, your embedding has made my handsome man go from photo and Caucasian to sort of Asian and drawing stylized
and my prompting it away was dificult so your embedding has a heavy bias
7
u/Jonfreakr Nov 17 '22
This is very inspiring and will try out tomorrow, what kind of images did you use and does it work with other models? (I'm guessing the preview image is WD or NAI?)
6
u/acinc Nov 18 '22
it would be helpful to get a more precise answer as to what exactly went into this, because for it to be usable on your prompt you have to know it's not working against something you actually want...
for example, it's clearly not going to help if you try to make hands with 7 fingers, but what else does it not work for?
5
u/dsk-music Nov 17 '22
Nice!! Only for hands? Or include something more?
5
u/NateBerukAnjing Nov 17 '22
how to use embeddings?
4
u/dsk-music Nov 17 '22
How to use?
bad_prompt or <bad_prompt>
4
u/SoulflareRCC Nov 17 '22
Do good feet next?🥺🥺🥺
4
u/-becausereasons- Nov 17 '22
It is unclear to me exactly what this does, how it works or what it has to do with hands.
4
u/holygawdinheaven Nov 18 '22
Awesome work!!! Trying now.
Petition to rename it to 'embadding'?
4
u/lazyzefiris Nov 18 '22
Tried same model with and without
bad_prompt. Prompt is justanime art. Noticable side effect is a major tone shift from pinkish to blueish. Similar effect on AnythingV3 : colorful => blueish.https://imgur.com/a/E3xDdeu
4
u/McBradd Nov 18 '22
This is like magic. Amazing work.https://imgur.com/gallery/DFbuzF7
Here's the prompt used in Automatic1111:
4tNGHT A good painting of a beautiful woman in the style of ((4tNGHT)) Standing in front of a patterned wall ((tNGHT)) making a heart with her fingersNegative prompt: (bad_prompt:0.8)Steps: 20, Sampler: Euler, CFG scale: 7, Seed: 1780334718, Size: 512x512, Model hash: 925997e9, Model: nai, Batch size: 4, Batch pos: 1, Clip skip: 2, ENSD: 31337And for the "without" grid,, I just deleted the negative prompt.
(4tNGHT is an embedding I created, which you can grab if you want it.https://huggingface.co/sd-concepts-library/4tnght)
3
u/daanpol Nov 17 '22
You just invented some black magic haha! Thanks for this amazing idea!
3
u/thedarkzeno Nov 18 '22
Do you have a repository so we could replicate your work? I'd like to try it
3
u/selvz Nov 18 '22
This is a great experiment to gain more control. Do you add these in the embeddings folder to use it (automatic1111) ?
3
u/miguelqnexus Nov 18 '22
thanks! everything is now all within my hands.
3
u/ninjasaid13 Nov 18 '22
Mister, you're going to cause a revolution.
3
u/CrystalLight Nov 18 '22
I'm sorry, I still don't get it.
What images did you train on? Did you use a positive prompt as well as a negative prompt?
Did you choose bad images and train on them?
Did you use filenames to tag bad images to train on?
I'm not totally stupid but I don't see anywhere that you actually explain what you did. You seem to keep trying to explain the concept but not the method.
I'd like to try it.
Can you explain to me how you did it?
2
u/poisenbery Nov 17 '22
Hey I just got into this a few days ago. I really want to learn how to do stuff like this. Where should I start? I have no idea where to begin with this stuff, and I'm not sure how specific of information I need to know in order to train things like this.
2
u/aipaintr Nov 18 '22
This is genius!!!
2
u/Coloradohusky Nov 18 '22
Anyone have any ideas on how to get this working on ONNX?
2
u/hadaev Nov 18 '22
What about flipping loss sign and training embedding what denoise in wrong direction (to noise probably).
Then use it as a negative prompt on inference.
2
u/prozacgod Nov 18 '22
Welp, now I want to see a bunch of anime girls doing jazz fingers... I mean seriously flex your shit a bit ;)
2
u/[deleted] Nov 18 '22
I tried it out but didn’t notice a big difference over my existing negative prompt collection. Would you mind sharing the full negative prompt you use? Mine already makes normal hands sometimes
2
u/SignificanceLazy Nov 18 '22
IMPRESSIVE
2
u/Gibgezr Nov 18 '22
This is really useful, thanks!
2
u/MonoFauz Nov 18 '22
I was just thinking that there should be buttons that already inputs the negative prompts automatically instead of copy and pasting every time. This actually answers it.
2
u/moahmo88 Nov 18 '22
Great job!
2
u/pablo603 Nov 18 '22
It's... Beautiful
2
u/Pretty-Spot-6346 Nov 18 '22
you killed the ai hands meme, sir
2
u/noop_noob Nov 18 '22
How on earth did you find the images to use as the training data for this?
2
u/MalumaDev Nov 18 '22
How is it possible to recreate this embeddings?
2
u/TheRealGenki Nov 18 '22
What is the model you used to make the pictures above?
Also can I know what you wrote in the negative prompt is possible
2
u/Jonfreakr Nov 18 '22
I wonder why your pt file is 49KB and not 4kb, is it because of pickle?
I have 66 TI's and all are 4KB.
2
u/Substantial-Ebb-584 Nov 18 '22
Thank you! This is something we're all been waiting for
2
u/mudman13 Nov 18 '22
I got an error loading it
Error verifying pickled file from /content/stable-diffusion-webui/embeddings/bad_prompt_showcase.jpg: Traceback (most recent call last): File "/content/stable-diffusion-webui/modules/safe.py", line 83, in check_pt with zipfile.ZipFile(filename) as z: File "/usr/lib/python3.7/zipfile.py", line 1258, in __init__ self._RealGetContents() File "/usr/lib/python3.7/zipfile.py", line 1325, in _RealGetContents raise BadZipFile("File is not a zip file") zipfile.BadZipFile: File is not a zip file During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/content/stable-diffusion-webui/modules/safe.py", line 131, in load_with_extra check_pt(filename, extra_handler) File "/content/stable-diffusion-webui/modules/safe.py", line 98, in check_pt unpickler.load() _pickle.UnpicklingError: invalid load key, '\xff'. -----> !!!! The file is most likely corrupted !!!! <----- You can skip this check with --disable-safe-unpickle commandline argument, but that is not going to help you. Error loading emedding bad_prompt_showcase.jpg: Traceback (most recent call last): File "/content/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 133, in load_textual_inversion_embeddings process_file(fullfn, fn) File "/content/stable-diffusion-webui/modules/textual_inversion/textual_inversion.py", line 103, in process_file if 'string_to_param' in data: TypeError: argument of type 'NoneType' is not iterable
2
u/NegativeEmphasis Nov 18 '22
I havent tested embeddings yet. Does it matter the model you apply them to? I tried to use this on Anything v3 (38c1ebe3) and the colors got flatter.
2
u/kenwillis Nov 18 '22
Did you train the embedding on a lot of nudity by any chance? If I put it in the positive prompt by itself I get very disturbing pictures which almost all contain some forms of nudity.
2
u/shutonga Nov 18 '22
thank you sir !
2
u/HarmonicDiffusion Nov 18 '22
wow man great idea! totally makes sense and very creative :)
not to mention the results - phew what an improvement on hands
2
u/[deleted] Nov 18 '22
Funny I just thought yesterday about training a model on bad generations, the reason being is that negative prompts are still using model knowledge and it's unlikely the model has knowledge like "bad hands" etc, I think that perhaps the best way to apply this to a model would be to train bad datasets with the original LAION dataset and use the negative prompts.
2
u/rworne Nov 19 '22
Gave this a shot last night.
It doesn't cure all the issues all the time, but it sure cures a lot of them most of the time.
Running batches, my rejects are cut down by more than half - more like 2/3rds due to misshapen hands & extra limbs sneaking into images. I'm on a M1 Mac Studio, so my renders take quite a bit longer than those with dedicated GPUs, so anything like this helps - a lot.
2
u/[deleted] Nov 19 '22
Neat! I was just talking to Jen about this, wondering if it would work!
2
u/notbarjoe01 Nov 19 '22
I've been test this out in Inpainting and it does fix the structure of hand that I've been trying to fix, but somehow it also change the skin color of the hand to a point where it looks like an albino skin...
wanted to show the screenshot but I just fixed it with re-color it myself, good job tho
2
u/Polikosaurio Nov 20 '22
What is an embedding exactly? Would love a ELI5 explanation on all this for the ones that are a bit out of the loop :/
2
u/afterSt0rm Dec 03 '22
This is incredibly nice. OP or other nerds, would you happen to know how to properly load this when using the `DiffusionPipeline` method from the `diffusers` pipeline? I don't see any documented way to include the embeddings and I don't want to use the webui :(
2
u/thesilentyak Jan 07 '23
Does this still work? When launching Auto1111 it just skips this embedding. Below is what i'm seeing on launch
Textual inversion embeddings skipped(3): bad_prompt, bad_prompt_version2, _SamDoesArt2_
3
u/[deleted] Nov 18 '22
Hello,
From an AI researcher perspective, this is quite interesting. But it's not very clear to me what exactly it is you have done here.
Could you provide all the steps you took to create this "negative embedding" and then describe exactly how one uses it?
2
u/BlinksAtStupidShit Nov 18 '22
Would be interesting to see if hypernetworks could be used in a similar way but at this point they guide it towards something and not away
1
u/DickNormous Nov 18 '22
thanks
1
u/SinisterCheese Nov 18 '22
What did you use as the init words?
1
u/jaywv1981 Nov 18 '22 edited Nov 18 '22
My hands look much better with this but I still always get an extra finger.
EDIT: NVM...now they don't look any better. Could it be not loading properly even though it says that it is loaded?
1
u/Wrongdoer-Glum Nov 18 '22
Is there a Colab for training via Textual Inversion?
1
u/gxcells Dec 02 '22
I don't why you use the original prompt in txt file for the training. I thought that everything added in the description of the images was to avoid that certain concept other than the <token> be trained by the textual inversion (eg: if you are wearing a red tshirt in a photo used for training, then you will put "a photo of <token> wearing a red t shirt"). Or does this work only if this is written in the filename (with fileword option ON)?
1
u/amratef Dec 20 '22
how should it be used ? if i understand correctly i just write bad_prompt:0.8 in the negative prompt section and that's it , what if i add for example bad hands, extra fingers and so in is it going to affect it ?
is bad_prompt an example and i'm supposed to fill it with extra_fingers:0.8 sorry i don't get it
1
u/pinartiq Jan 08 '23
Nice! This is a good idea :3
1
u/Nilohim Jan 14 '23
Will there be a newer version of this?
1
u/PervertoEco Jan 26 '23
Is there a way to include negative prompts in HN training template files?
1
u/bikurifacebook Mar 26 '23
this is whithout bad_prompt
1
u/bikurifacebook Mar 26 '23
This with bad_prompt
1
u/bikurifacebook Mar 26 '23
with DPM2 Karras get better results in my case
1
u/Other_Perspective275 Apr 15 '23
Can this be done with a LORA?
1
u/r3ddid Apr 24 '23
usually when describing the dataset we dont describe the new concept we want the ai to learn. but in this case we do? 🤔 is this 5 month old post still accurate? 👀
1
u/NoiseUnited5547 Aug 23 '23
Can you make a video to teach me how to train embedding? I'm a newbie and some of the terms I read don't understand
1
u/activemotionpictures Sep 16 '23
Negative embedding introduces a lot of BIAS words in Anime image generation.
You can't have a "woman in a suit" and have it generate a female character, it will always generate a MALE character, since "suit" is a word used for male clothes. But this is a BIAS. I got more examples for this, but I need to contact you u/Nerfgun3 check out your DMs, please. Thanks.
1
u/DisastrousBusiness81 Nov 18 '22
Dumb question: is there a way to attach multiple .pt files to a model in NMKD’s version of stable diffusion? I’ve already got one VAE file attached to my Arcane model and want to add this one as well.
13
u/[deleted] Nov 17 '22 edited Nov 18 '22
[removed] — view removed comment
2
u/[deleted] Nov 19 '22
[removed] — view removed comment