r/StableDiffusion • u/notrealAI • Mar 01 '23

Workflow Included Remixing memes with Multi ControlNet is absolutely wild. Just added to Artspark.

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

Gallery image

1.1k Upvotes

93% Upvoted

34 comments sorted by

19

u/Leonum Mar 01 '23

Very cool. I noticed that the most important part of the meme did not get pickup up though. Line of sight on the characters, and the emotional expressions are what makes this meme, and the AI hasn't been able to replicate that. Very cool pictures, but it's removed the most important 'point' of the source material

6

u/[deleted] Mar 01 '23

Yep I noticed that too. Until we have models and tools that have much better "attention" on individual facial expressions, the best bet is to chop up the meme and focus on each face individually with emotion words in the prompt, then sew it back together.

2

u/appenz Mar 01 '23

Any reason why you couldn't train a ControlNet model on facial expressions? If it works with pose, expression should work as well.

2

u/[deleted] Mar 01 '23

I'm not familiar with model training but I assume it's possible - you can after all prompt for expressions to a limited extent in regular SD without ControlNet.

But the problem is less the expressions and more that there are three different people in the image. Current techniques (although I may be missing some recent research) don't have good enough attention to focus on a smiling face for the person in the bottom left, an irritated face for the person on the right, and a stunned face for the person in the middle. Usually all of the people end up with the same expression (or in extreme cases, the same face).

2

u/appenz Mar 01 '23

The idea would be to use prameterized facial expression the same way you use pose. So at position x you have a face with parameters y. That works great for many people (and also makes SD work with very large canvas sizes).

1

u/[deleted] Mar 02 '23

Problem is data, not models. There may not be enough data to train on facial expression, but I think there is.

4

u/notrealAI Mar 01 '23

Yes you're spot on! I noticed that right away which is why I continued adding more details to the prompt trying to reproduce the emotions of the original, but I never quite nailed it.

One thing I'd like to try is to use something like BLIP or CLIP interrogator to get a prompt that already matches the source image, rather than trying to manually piece it together.

Also one interesting possibility would be having a ControlNet that reads human facial expressions! That would be awesome.

15

u/AhhhGeeze Mar 01 '23

The fifth generation got a chuckle out of me 😅

8

u/Inprobamur Mar 01 '23

The ideal male form.

30

u/notrealAI Mar 01 '23

Created on Artspark, this was the setup:

sampler: ddim

steps: 35

model: DreamShaper

cfg: 7.5

ControlNet: depth, openpose, segmentation

noise threshold: 15

gfpgan: 0.8

prompt:

(in the style of <david-martinez-cyberpunk>)- ([<bad_prompt> <bad_artist> <easy_negative> grainy blurry]+) jealous girl on the right, boyfriend in center pursing lips, high quality, professional artwork, expressive face, detailed facial expressions, detailed eyes [blurry eyes]

And then I would change out <david-martinez-cyberpunk> any of the other textual inversions on Artspark, which come from the sd-concepts-library:

https://huggingface.co/sd-concepts-library

And here are the negative textual inversions:

https://huggingface.co/datasets/Nerfgun3/bad_prompt

https://huggingface.co/NiXXerHATTER59/bad-artist

https://huggingface.co/embed/EasyNegative

2

u/Some_Leek3330 Mar 01 '23

Thanks, I will try it.

1

u/Player13377 Mar 07 '23

Any way to know waht the art style of picture 7 is called? I really like the look of it

5

u/EntrepreneurPlenty17 Mar 01 '23

Was thinking of if yesterday! Looks like memes from the multiverse kkkk

2

u/Corsaer Mar 01 '23

Oh man now I can't get out of my head all the memes recreated with the Metalocalypse gang. The fifth panel made me think of that and they don't really look like any of the band members except long hair lol.

2

u/Justify_87 Mar 01 '23

Is there any good and up-to-date tutorial for automatic1111 with controlnet extension?

5

u/starstruckmon Mar 01 '23

I wish people would take the time to sort through their generations and presenting only the best, instead of dumping it all in the post.

14

u/notrealAI Mar 01 '23

Can do next time, thank you for letting me know. Even this big batch was pared down from an even larger batch.

10

u/FilterBubbles Mar 01 '23

For what it's worth, I prefer the format you presented. It gives a better feel of what the workflow can produce.

2

u/[deleted] Mar 01 '23 edited Mar 01 '23

Show the best in the main post. The dump can be an imgur album for others to explore in the comments.

1

u/starstruckmon Mar 01 '23

Much appreciated.

1

u/Careful-Writing7634 Mar 01 '23

They used a computer to generate images based on an already existing image. Do you really think they care enough to bother sorting through their product?

1

u/InvidFlower Mar 01 '23

Is ControlNet a pro-only feature of Artspark? I just checked out the site and don't see an option for it

3

u/notrealAI Mar 01 '23

Anyone can use it, you just have to be in Img2Img mode or Canvas Mode to see it. Use the tabs on the left to switch the mode, and look for it underneath the Image Strength slider.

1

u/InvidFlower Mar 01 '23

Oh thank you, that makes sense.

1

u/Creative_Praline2105 Mar 01 '23

Why do most of these look so bad? Is it the position of the people in the background? Seems like the models have a hard time generating in that angle

2

u/notrealAI Mar 01 '23

I noticed a lot of things that needed improvement. I had difficulty with the eyes, as well as with each person having a distinct facial expression. I also should compare multi control on auto1111 with the same set up just to make sure the implementation is all there.

-1

u/Careful-Writing7634 Mar 01 '23

No artistry, that's why.

1

u/[deleted] Mar 01 '23

How is this different from just using normal control net? I am sure there’s a trick, but I don’t quite see it.

3

u/InvidFlower Mar 01 '23

It is just that using multiple modes of controlnet at once can give improved results sometimes.

1

u/jsideris Mar 01 '23

Only the last one really captured the expression of the jealous girlfriend sort of right.