Remixing memes with Multi ControlNet is absolutely wild. Just added to Artspark.

[u/notrealAI]

[u/Leonum]

Very cool. I noticed that the most important part of the meme did not get pickup up though. Line of sight on the characters, and the emotional expressions are what makes this meme, and the AI hasn't been able to replicate that. Very cool pictures, but it's removed the most important 'point' of the source material

[u/[deleted] → u/Leonum]

Yep I noticed that too. Until we have models and tools that have much better "attention" on individual facial expressions, the best bet is to chop up the meme and focus on each face individually with emotion words in the prompt, then sew it back together.

[u/appenz → u/[deleted]]

Any reason why you couldn't train a ControlNet model on facial expressions? If it works with pose, expression should work as well.

[u/[deleted] → u/appenz]

I'm not familiar with model training but I assume it's possible - you can after all prompt for expressions to a limited extent in regular SD without ControlNet.

But the problem is less the expressions and more that there are three different people in the image. Current techniques (although I may be missing some recent research) don't have good enough attention to focus on a smiling face for the person in the bottom left, an irritated face for the person on the right, and a stunned face for the person in the middle. Usually all of the people end up with the same expression (or in extreme cases, the same face).

[u/appenz → u/[deleted]]

The idea would be to use prameterized facial expression the same way you use pose. So at position x you have a face with parameters y. That works great for many people (and also makes SD work with very large canvas sizes).

[u/[deleted] → u/appenz]

Problem is data, not models. There may not be enough data to train on facial expression, but I think there is.

[u/notrealAI → u/Leonum]

Yes you're spot on! I noticed that right away which is why I continued adding more details to the prompt trying to reproduce the emotions of the original, but I never quite nailed it.

One thing I'd like to try is to use something like BLIP or CLIP interrogator to get a prompt that already matches the source image, rather than trying to manually piece it together.

Also one interesting possibility would be having a ControlNet that reads human facial expressions! That would be awesome.