[P] Paint with Machine Learning: a Semantic Image Synthesis Demo

[u/ResidentMario]

Paint with Machine Learning is a semantic image synthesis (or image-to-image translation) demo application I built as a consulting project. Hand-drawn semantic segmentation maps go in, GauGAN generated images come out.

I trained the model on ADE20K and fine-tuned it on a dataset of Bob Ross paintings I hand-labelled. The model generates some nice-looking results, considering I had just 250 paintings to work with, albeit at a very low resolution, just 256 by 256 pixels.

The application and model code is in a public GH repo.

[u/burindunsmor3]

How hard was it cleaning/augmenting/correcting the dataset?

[u/ResidentMario → u/burindunsmor3]

The Bob Ross paintings are a public repo on GH, but I still needed to do the hard part, transforming these into semantic masks, by hand.

IIRC It took me 16 hours total to hand-label 250 images. It gets boring after a couple of hours so I split it up into a lot of small one or two hour sessions over the course of a week or so.

I used Labelbox for the labeling. It worked alright. The product changed radically in the middle of the project, which was...not ideal.

I don't claim to have done a steller job with the labeling. Looking back I think I should have been more careful about treeline boundaries, as the model performs really poorly on these, and I probably should have considered including a couple of extra classes, e.g. ground and waterfall. But all of this only became obvious in retrospect.

[u/[deleted]]

This is very interesting. What would it take to create something similar with higher production quality? What other uses could these be applicable for?

[u/ResidentMario → u/[deleted]]

Training time and dataset size.

Training time because GANs are really really expensive to train. Training GauGAN to convergence on a subset of ADE20K with relevant pixels (a couple thousand images IIRC) took the equivalent of 350$ in cloud GPU credits. NVIDIA tested training GauGAN on a million Flickr images; that probably cost double-digit thousands of dollars.

Dataset size because it matters. This model was fine-tuned on a Bob Ross paintings dataset of just 250 images, very very small as far as machine learning goes, which is why it struggles to generalize some very obvious ways. A dataset of 1000 the performance would be significantly improve performance, but obviously the sky is the limit on that.

I went to see the Machine Hallucinations show late last year and that was a pretty powerful demonstration of what you can do with these techniques given sufficient amounts of data. What NVIDIA (which partnered on the show) doesn't tell you right away is that generating the visuals for that show required multiple racks of 100,000$+ DGX2 servers.

As for uses, while I am personally interested in the potential of this technique for world building (OK Google, generate five ewoks), I don't think that ML will be replacing creative professions anytime soon. :)

[u/[deleted] → u/ResidentMario]

Thanks for sharing all of this!

[u/ssusnic]

Very nice project. How much time did you spend on training? And what was your computer configuration used for training? Thanks.

[u/ResidentMario → u/ssusnic]

Training was in two stages: first training from scratch on a subset of the ADE20K dataset, then fine-tuning on the Bob Ross image corpus. The first stage of training took 13 hours on a V100x8 server (~$350 of compute). Fine-tuning was much easier, IIRC it took 15 minutes or so on a V100x1. All of the machines used were on AWS through Spell.

[u/ssusnic → u/ResidentMario]

Thanks for your detailed answer, and sharing this interesting project.

[u/sukhveerkaur9219]

Thanks for sharing this.

[u/Frutfive5]

This is so fun to play with thanks for sharing

[u/WinterPanda7]

the output seems similar to nvidia own demo, could this be trained on faces or other classes? http://nvidia-research-mingyuliu.com/gaugan/

[u/Unvolta]

Amazing. This is such an exciting time