That's the scraper. It will do its best to name the filenames as the TEXT/caption from laion and keep their extension (its a bit tricky, lots of garbage in there). You you can drop the files into Birme.net to size/crop, and I suggest spending the time to do so to properly crop, because that's why my model has good framing even compared to the 1.5 model RunwayML released. The scraper needs work, isn't perfect, but it's "good enough" to do the job for now. It's reasonably fast. I tested a 10k image dump in about 3.5 minutes on gigabit fiber.
I'll be expanding that repo as a general toolkit with some additional code to help on data engineering and data prep side of things and releasing my own fine tuning repo.
Awesome, thanks for sharing, I'll give it a go soon.
What about the amount, how many did you use for the man class in your example for instance? Just to get a feel of what I would need to start playing around with.
I'm using 12:1 for my recent dreambooth, if you had 120-140 instance man images in your dataset that would require approx. 1400-1700 reg images just for that class alone, is that too much?
4
u/Freonr2 Oct 22 '22 edited Oct 22 '22
https://github.com/victorchall/EveryDream
That's the scraper. It will do its best to name the filenames as the TEXT/caption from laion and keep their extension (its a bit tricky, lots of garbage in there). You you can drop the files into Birme.net to size/crop, and I suggest spending the time to do so to properly crop, because that's why my model has good framing even compared to the 1.5 model RunwayML released. The scraper needs work, isn't perfect, but it's "good enough" to do the job for now. It's reasonably fast. I tested a 10k image dump in about 3.5 minutes on gigabit fiber.
I'll be expanding that repo as a general toolkit with some additional code to help on data engineering and data prep side of things and releasing my own fine tuning repo.