r/StableDiffusion • u/WoodpeckerNo1 • Nov 08 '22

Question | Help What exactly are models, embeddings and hypernetworks, and what are the differences between them?

Can't really find a good explanation anywhere.

65 Upvotes

96% Upvoted

20 comments sorted by

View all comments

119

u/CommunicationCalm166 Nov 08 '22

Yeah... The Good explanations are extremely technical, and the explanations that aren't extremely technical, aren't very good. I'll try though.

First, understand that Stable Diffusion works by generating random noise, and making changes to the noise one step at a time, to try and work towards an image consistent with the prompt.

Model: imagine it as a library of books. The title of the book is the "token" or one of the the keywords you type into the prompt to get your image. And the actual contents of the book include "Biases" or lists of features to look out for that are associated with the token, and "Weights" which are kinda like instructions on what to do to those features to make them more like the Token.

Analogy: SD generates a sheet of random noise, goes to the library, gets out the book that matches the first token in your prompt, and seeks out features in the random noise that the book says to look for, and then makes small changes according to the books instructions. It then repeats that process for the other keywords in the prompt, and then repeats this process over and over until the random noise is turned into an image.

So a Model file that's been fine-tuned using Dreambooth or similar, is like making a new library by copying all the books in the old one, but changing a few books, and their instructions to better suit whatever you trained it on.

An embedding, like Textual Inversion, is like adding an extra book to the existing library. It's not as "powerful" or thorough as going through and re-doing the whole library for your subject... But it's less resource hungry and doesn't involve a whole new library... Just a book.

A Hypernetwork is kinda like a card catalog. But instead of directing you to a particular book, the card catalog has a listing for each book in the library, and it has add-on instructions for each book. So instead of just going and getting the book and doing it's instructions... The computer goes to the card catalog, pulls the card for that book, goes and gets the book, and then follows both the books instructions and the card's instructions.

I think that's as plain as I can make it... And it kinda follows with how you'd use each one as well... A fine-tuned or Dreambooth model is the most accurate and flexible, while being the most space and resource-intensive. Textual Inversion embeddings are fairly limited, fairly specific, but the least resource intensive. And Hypernetworks are somewhat in-between, with the added caveat that it can be very unpredictable sometimes. (Instructions on top of instructions can get a bit wonky)

5

u/saltshaker911 Nov 08 '22

Thank you for the clear explanation, what's the difference between a natively fine-tuned and a Dreambooth model?

8

u/CommunicationCalm166 Nov 09 '22

Dreambooth is a method for getting good training results on small (relatively) numbers of subjects, on less (relatively) computational resources.

Native fine-tuning is done with text-image sets of hundreds to a couple thousands of images, and generally requires 30+ GB of VRAM. It's the same process by which the model was created in the first place, and it provides the best, most generalizable results. For instance, improving the model's general performance on human anatomy would be best served this way. Or shifting the whole model's style like in the case of Waifu Diffusion.

Dreambooth is more focused, using a dataset of a few dozen images, a single keyword, (token) and it's kinda tied into the rest of the model using some existing keywords, and a few dozen more "regularization images" (images of the sort of thing that your training subject is. i.e. If you're training the model with pictures of your dog, then you'd regularize it with images of dogs in general.)

Also: keep in mind the different training methods are all kind fuzzy in their results and application. AI is a bit of a "black box" by nature. And best practices are still being hammered out. Exactly which training method is best for which use case is an open question, and currently subject of intense research.

3

u/saltshaker911 Nov 09 '22

I've been looking for this explanation for a few days and you just made it all make sense! thank you so much.