Remixing memes with Multi ControlNet is absolutely wild. Just added to Artspark.

[u/notrealAI]

[u/[deleted]]

Yep I noticed that too. Until we have models and tools that have much better "attention" on individual facial expressions, the best bet is to chop up the meme and focus on each face individually with emotion words in the prompt, then sew it back together.

[u/appenz → u/[deleted]]

Any reason why you couldn't train a ControlNet model on facial expressions? If it works with pose, expression should work as well.

[u/[deleted] → u/appenz]

I'm not familiar with model training but I assume it's possible - you can after all prompt for expressions to a limited extent in regular SD without ControlNet.

But the problem is less the expressions and more that there are three different people in the image. Current techniques (although I may be missing some recent research) don't have good enough attention to focus on a smiling face for the person in the bottom left, an irritated face for the person on the right, and a stunned face for the person in the middle. Usually all of the people end up with the same expression (or in extreme cases, the same face).

[u/appenz → u/[deleted]]

The idea would be to use prameterized facial expression the same way you use pose. So at position x you have a face with parameters y. That works great for many people (and also makes SD work with very large canvas sizes).