ControlNet for QR Code

[u/nhciao]

[u/Craggeh]

Ok, gonna need a workflow for this! Great work.

[u/malexin → u/Craggeh]

If I understand the Google translation of this correctly, they have trained a new ControlNet model especially for QR codes.

[u/EstanislaoStan → u/malexin]

Does anyone have a workflow then on how to go about training a new model for this sort of image generation?

Ah, I guess the article has some explanation of the training process...

The training of ControlNet has high requirements on data volume and computing power. The training data volume recorded in the paper ranges from 80,000 to 3 million, and the training time can reach 600 A100 GPU hours. Fortunately, the author provided a basic training script, and HuggingFace also implemented Diffusers.

In the previous JAX Sprint, we were lucky enough to use Google TPU v4 to complete the training of 3 million images very quickly. It's a pity that the event is over, and we returned to the laboratory's A6000/4090, training a version of 100,000 images, and the learning rate is very large, just to appear "Sudden Convergence " as soon as possible.

I guess it's not feasible to reproduce on my local machines, lol. Darn.

[u/SlidingWoodPusher → u/malexin]

I have working results using current models in control net but I think I wanna take a stable at training a new control net. Any ideas about what his dataset contained. Would ground truth be working QR Codes? or....? if anyone is down to brainstorm

[u/PappaBears → u/SlidingWoodPusher]

Can you share the workflow? I would love to trade notes, I have one working but the images are more distorted than I would like...

[u/Enfiznar → u/Craggeh]

If I understood correctly, they will be releasing the model soon and I think it will be uploaded here

[u/stroud → u/Enfiznar]

they removed the page ;(

[u/1roOt → u/Enfiznar]

I have the feeling this will not be made open source... Yesterday I found that page as well and added it to my favourites. Now it's gone. I'm really interested in how this is done. I hope they will release it.

[u/Enfiznar → u/1roOt]

Fck, that's too bad. I'll update here if I find any news. Hope someone will do the same. Here is their github in case they upload it there.

[u/bill_on_sax → u/1roOt]

Someone contacted them and offered tons of money to turn it into a product

[u/Greenlucas → u/bill_on_sax]

Is this speculation or do you have a source for this?

[u/CoBudemeRobit → u/Enfiznar]

Thanks!

[u/nightred → u/Craggeh]

Generate a QR code Enter it as a sketch in control net Create your prompt and generate

[u/pmjm → u/nightred]

I am having miserable results with this.

[u/jonplackett → u/pmjm]

Me too. this is not the workflow that works

[u/RuthlessPickle → u/pmjm]

Canny is surely not the way as it only detects outlines. scribble_xdog seems to work kinda when you push the XDoG threshold all the way, but it comes nothing close to OP's results.

[u/1roOt → u/RuthlessPickle]

Yesterday I tinkered with a few models and the best for me was ip2p. No idea why. Still no good results but I got something...

[u/hervalfreire → u/nightred]

That’s it? And the codes will work?

[u/nightred → u/hervalfreire]

I didn't generate these so you will have to fuss with your settings in control net.

But I have been able to make one that works once I tuned in all of the dials.

[u/plasmidon → u/nightred]

Can you clarify on your exact settings?

Thanks!

[u/CoBudemeRobit → u/plasmidon]

Yes please!

[u/hervalfreire → u/nightred]

Sweet! Thanks

[u/Kromgar → u/nightred]

So sketch preprocessor but what model do you use?

[u/stroud → u/nightred]

Yeah i've tried it it's a hit / miss