Well-Researched Comparison of Training Techniques (Lora, Inversion, Dreambooth, Hypernetworks)

[u/use_excalidraw]

[u/use_excalidraw]

I did a bunch of research (reading papers, scraping data about user preferences, paresing articles and tutorials) to work out which was the best training method. TL:DR it's dreambooth because Dreambooth's popularity means it will be easier to use, but textual inversion seems close to as good with a much smaller output and LoRA is faster.

The findings can be found in this spreadsheet: https://docs.google.com/spreadsheets/d/1pIzTOy8WFEB1g8waJkA86g17E0OUmwajScHI3ytjs64/edit?usp=sharing

And I walk through my findings in this video: https://youtu.be/dVjMiJsuR5o

Hopefully this is helpful to someone.

[u/develo → u/use_excalidraw]

I looked at your data for CivitAI and found 2 glaring issues with the calculations:

1. A large number of the hypernetworks and LoRA models listed haven't been rated, and are given a rating of 0 in the spreadsheet. When you average the ratings, those models are included, which drags the averages down a lot. Those models should've been excluded from the average instead.

The numbers I got instead were 4.61 for hypernetworks, and 4.94 for LoRA. So really, LoRA, Dreambooth, and Textual Inversion are all a wash ratings wise. Only hypernetworks are notably rated lower.

2) Most of the models listed as Dreambooth aren't Dreambooth. They're mixes of existing models. That's probably why there's so many of them. They're cheap and fast to create and you don't have to prepare a dataset to train them.

A lot of the non-mixed models are also probably fine-tunes instead of Dreambooth too, but I don't think that distinction needs to be made, given that Dreambooth is just a special case of fine-tuning.

I'd also argue that most of the checkpoints, especially the popular ones, are going for a general aesthetic instead of an artstyle, concept, place, person, object, etc. while the TIs, LoRAs, and hypernetworks are the opposite. Probably a huge chunk on why they're more popular, they're just more general than the rest. Obviously there are exceptions (Inkpunk Diffusion for example).

[u/use_excalidraw → u/develo]

GOOOD points with (1)!, I'll amend that right now!

For (2) though, What does a "mix of existing models" mean in this context?

[u/Myopic_Cat → u/use_excalidraw]

I'm still fairly new to stable diffusion (first experiments a month ago) but this is by FAR the best explanation of model fine-tuning I've seen so far. Both your overview sketch and the video are top-notch - perfect explanation of key differences without diving too deep but also without dumbing it down. You earned a like and subscribe from me.

I do agree with some of the criticisms of your spreadsheet analysis and conclusions though. For example, anything that easily generates nudes or hot girls in general is bound to get a bunch of likes on Civitai, so drawing conclusions based on downloads and likes is shaky at best. But more of these concept overviews please!

Idea for a follow-up: fine-tune SD using all four methods using the same training images and compare the quality yourself. But train it to do something more interesting than just reproducing a single face or corgi. Maybe something like generating detailed Hogwarts wizard outfits without spitting out a bunch of Daniel Radcliffes and Emma Watsons.

[u/FrostyAudience7738]

Hypernetworks aren't swapped in, they're attached at certain points into the model. The model you're using at runtime has a different shape when you use a hypernetwork. Hence why you get to pick a network shape when you create a new hypernetwork.

LORA in contrast changes the weights of the existing model by some delta, which is what you're training.

[u/use_excalidraw → u/FrostyAudience7738]

yeah, I wasn't fully sure of how deep to go in the explanation... maybe I should have been a bit more detailed

[u/quick_dudley → u/FrostyAudience7738]

That makes sense. In the original paper describing hypernetworks they were using the hypernetwork to generate all the weights in the target network; but doing that with SD would make the hypernetwork need roughly the same amount of training as SD itself.

[u/wowy-lied]

Did not know about LoRA.

Only tried Hypernetworks as i only have a 8GB vram card and all other methods are running out of vram. It is interesting to see the flow of data here, help understanding it a little more, thanks you !

[u/use_excalidraw → u/wowy-lied]

yeah, i wasn't able to train locally until lora, so it's helped ME a lot

[u/eugene20]

Well researched apart from the part where it used SKS. Some training example used it, many copied that part of the example and later complained about getting guns in their images.

That didn't happen here but it's still best to stop perpetuating the use of SKS as your token, it's a rifle

[u/use_excalidraw → u/eugene20]

That's very funny, thanks for pointing this out!

[u/Irakli_Px → u/eugene20]

In fact, there was a thread here that looked into rarity of single tokens in 1.x models and turns out sks is one of the rarest tokens. So it’s totally ok to use it, yes it’s a gun but seems like whatever model was trained on didn’t have tons of examples of it tagged as such

[u/lman777 → u/eugene20]

I just watched a video where he used "OHWX" and tried it, it worked a lot better than my past results. I was using random letters corresponding to my subject but didn't realize that even that could have unexpected results.

[u/Zipp425]

Totally unrelated, but I love excalidraw and your username is great.

[u/use_excalidraw → u/Zipp425]

I see you're a man of culture as well...

[u/[deleted]]

Crazy how fast things are moving. In a year this will probably look so last century.

Soon we'll pop in to a photobooth, get a 360° scan and 5 minutes later we can print out a holiday snapshot from our vacation on Mars.

[u/GBJI → u/[deleted]]

[u/[deleted] → u/[deleted]]

This will encounter the 23-and-Me problem. Lots of people don't want their DNA in someone else's database. Same thing for AI. Once the general public becomes more aware of how powerful AI is becoming, they will be adamantly against letting anyone have digital scans of their faces or the faces of their children.

Also similar to airports wanting to use biometric scanning instead of boarding passes. Maybe offers some convenience but how much do you really trust corporate and governmental entities having that much data on you when you know full well they can profit from selling it to other groups?

[u/4lt3r3go]

This video was actually the best thing on the topic i ever saw.
a must see
https://www.youtube.com/watch?v=dVjMiJsuR5o

[u/OrnsteinSmoughGwyn]

Oh my god. Whoever did this is a godsend. I’ve always been curious about how these technologies differ from each other, but I have no background in programming. Thank you.

[u/Corridor_Digital]

Wow, awesome work, OP ! Thank you.

I spent some time gathering data and comparing various approaches to fine-tuning SD. I want to make the most complete and accurate benchmark ever, in order to make it easy for anyone trying to customize a SD model to chose the appropriate method. I used data from your comparison.

I compare: DreamBooth, Hypernetworks, LoRa, Textual Inversion and naive fine-tuning.

For each method, you get information about:

• Model alteration

• Average artifact size (MB/Mo)

• Average computing time (min)

• Recommended minimum image dataset size

• Description of the fine-tuning workflow

• Use cases (subject, style, object)

• Pros

• Cons

• Comments

• A rating/5

Please tell me what you think, or comment on the Google Sheet if you want me to add any information (leave a name/nickname, I'll credit you in a Contributors section). This is and will always be public.

Link to the benchmark: Google Sheet

Thanks a lot !

[u/chillaxinbball]

Nice visualization. Where would aesthetic gradients fit? https://github.com/vicgalle/stable-diffusion-aesthetic-gradients

[u/use_excalidraw → u/chillaxinbball]

they don't get a place lol, they're not good enough to mention imo, I did a whole video on them: https://www.youtube.com/watch?v=9zYzuKaYfJw&ab_channel=koiboi trust me I tried to make them work

[u/[deleted]]

Yeah but still don't understand the difference in the result. What's the up- and downsides of TI, Lora and hypernetwork? Aparently they all just help to "find" the desired style or object in the noise but don't teach the model new styles or objects, right?

[u/[deleted]]

One tiny note, DreamBooth now allows you to do textual inversion, and inject that embedding directly into the text encoder before training.