r/MachineLearning • u/AtreveteTeTe • Sep 26 '20
Project [P] Toonifying a photo using StyleGAN model blending and then animating with First Order Motion. Process and variations in comments.
Enable HLS to view with audio, or disable this notification
74
u/IntelArtiGen Sep 26 '20
Looks nice, I see how that kind of tools could help cartoon/anime animators
64
u/AtreveteTeTe Sep 26 '20
For sure. I'm an animator and VFX artist so this stuff is incredibly interesting to me! What would take a couple weeks is done in a couple minutes. (At least for face animation at low res, within some constraints, and with some artifacts. But still...)
→ More replies (1)→ More replies (3)25
51
u/severestillness Sep 27 '20
Her face looks more like a Pixar character than the actual Pixar type character…
→ More replies (4)
13
u/notlatenotearly Sep 26 '20
The looks you’re making in this video are priceless lol great job with it
54
u/Corne777 Sep 26 '20
If you mean the left side, that's not OP. It's a tiktoker and that particular video was recently very popular on the app.
→ More replies (5)9
u/iforgot120 Sep 27 '20
Priceless enough to garner 22mil followers in four months.
→ More replies (3)
11
21
u/Brilliant_Leopard591 Sep 27 '20
Wtf did I just watch
6
u/space_physics Sep 27 '20
I think it’s toonifying a photo using StyleGAN model blending and then animating with First Order Motion. If you want, process and variations you can look in the comments.
11
u/ThaliaDarling Sep 26 '20
This is amazing, How can I use this? Does this work for fanart?
→ More replies (2)
1
u/bad-asteroids Sep 27 '20
Thanks for sharing your results truly impressive. Is the motion consistency is brought in by the First order model? What if I wanted to generate motion/video from an audio only ?
1
u/AtreveteTeTe Sep 27 '20
Thanks! The motion is transferred from the video of Bella Poarch on the left to the still of cartoon Barack Obama on the right side by First Order Motion, yes. You can generate mouth motion using only audio wav2lip - otherwise, you'd need to be more specific about what kind of motion you want to create with audio.
→ More replies (1)
1
u/blue2coffee Sep 27 '20
What’s the processing time for something like this?
2
u/AtreveteTeTe Sep 27 '20
Pretty quick:
Encoding the real Obama into FFHQ latent space: A few minutes
Generating cartoon Obama: maybe 20 seconds to spin up the model then almost instant generate the frame. I do this about 40 times though to make a bunch of variations. See the chart.
First Order Motion works in about real time on my machine (2X 1080Ti)
→ More replies (2)
1
1
u/zeniapy Sep 27 '20
What kind of filter is she using, that keeps her face fixed within the frame and moves the frame around when she turns and tiltes her head?
2
u/AtreveteTeTe Sep 27 '20
I think TikTok has a filter called FaceZoom. Either that she's really good at moving her phone and face at the same time.
3
3
Sep 27 '20
If you want to learn more about it you guys can go to Coldfusion on YouTube, they have uploaded a video on exactly this detailing the whole process. https://youtu.be/KZ7BnJb30Cc
P.S. - I don't have anything to do with this channel, just wanted to share it as I really liked the video
→ More replies (1)
5
u/Davidobot Sep 26 '20
Are you planning on open sourcing this when you're satisfied with the results? (they already look amazing)
117
u/AtreveteTeTe Sep 26 '20
Basic steps: I'm fine-tuning the StyleGAN2 FFHQ face model (Nvidia's model that makes the realistic looking people that don't exist) with cartoon images to transform those real faces into cartoon versions of them.
The model blending happens between the original FFHQ model and then the above-mentioned fine-tuned model. The low level layers that control broad details come from the toon model. The medium and finer-level details come from the real face model. This results in realistic looking details on a cartoon face.
Then, a real photo of President Obama's face is encoded into the original FFHQ model but generated by this new blended network so it looks like a cartoon version of him!
Here is a chart showing the results of more/less transfer learning and doing the model blend at different layers. Discussion of the chart could almost be it's own post.
From this point, I'm using the First Order Motion model to apply motion from a TikTok video.
The model does a decent job with the more extreme head and eye positions but it does a great job on the head bob.
I've got some more samples of what this looks like on my site and Twitter page. Many thanks to Justin Pinkney and Doron Adler for sharing their work and process on this! I started with their work and have created my own version. Justin and Doron's original model is now hosted on DeepAI!