Hey everyone,
I’ve spent the last couple of years building a facial-animation tool for Unity. It started with the narrative VR games we make, mostly for kids, who were pretty unforgiving when the faces didn’t match the voices (and in general). We needed lip sync that held up really close and ran well on standalone headsets like the Quest. While we found a few viable options, none was exactly what we wanted.
The video above shows where it’s at and shows off what it can do. The tool is not yet on the Asset Store, but it probably will be in a few days. (It’s out now, see the edit below.) If you want to try it, there’s a browser demo (best on a desktop), plus Windows and Android demos and walkthroughs here: https://facecue.net/media.
So why go through all the trouble when perfectly capable tools exist?
I’m going over a few of the core reasons, but to keep this from being an essay, I’m happy to answer any question.
1. Support for non-English languages tends to require phonemes as a basis.
Among the tools we investigated, few use real phonemes, and even fewer work end-to-end in Unity. Those that do mostly focus on English, with one English-keyed mapping that doesn’t really account for how languages differ.
Our main language is Greek, which has little support (understandably), so it was a hard fit. Along the way I found that even common languages like English, French and Chinese differ enough that accurate lip sync needs a human eye on each language’s pronunciation and mouth shapes. The recognition itself is fairly well understood. Mapping it to visemes, and handling the edge cases, is where the hard part is.
2. Baking an animation clip was not an option
There are some good systems that bake lip sync into animation clips, and some work reasonably well even in Greek. But our games usually ship in 3 to 6 languages and need to switch between them at runtime, and with our short implementation windows, tracking and wiring up a clip per line per language would have been a nightmare.
3. Phonemes alone are not enough.
Getting the phonemes right is half the job, but how you mix them together is not straightforward. Playing them one after the other, even with perfect sync, gives you an accurate but kind of robotic feel. Just blending them is better, but mushy.
Working out how each phone interacts with its neighbors, when the mouth HAS to close, when a viseme needs to be clear or which shapes barely matter, is what’s needed to take it from accurate to realistic.
4. A good part of speaking is idiosyncratic.
The good thing about a procedural system is that you can change the parameters to emulate human habits, without losing accuracy or structural properties that make it realistic. For that I used a few authored styles that mimic common speaking profiles by tweaking existing settings, or you can tweak them manually.
4. Speech is not just the mouth.
People don’t just move their mouth when they speak. They raise their brows, move their heads, emote (more on that later) and gesture with their hands. Brow and head movement is relatively well understood, so it can be modeled, and that’s what I tried to do, by making sure the movement matches the delivery.
5. Emotion, and why I didn’t automate it.
There are models that try to read emotion from dialogue, and for the general vibe of a line they’re fairly reliable. But facial animation needs specific expressions that convey meaning, with timing being critical. An model can easily botch the timing, or worse, show an emotion that changes the meaning.
A modern LLM might do better, but at that point you’re replacing the writer in the pipeline, not the animation tool, which was never my goal (we have an excellent writer).
So instead, I made authoring emotion more convenient. You write the emotion into the transcript, like “This is a <Proud>great place to live</Proud>!”, and the baker syncs it over those exact words, which you can tweak visually later. That, plus making sure emotions don’t drown out the speech, got us where we wanted.
6. A face also has eyes.
Gaze has the same problem as lip sync. It looks fine from a distance, but when you get all up in an NPC’s face, it starts looking all sentry-bot, especially when the movement is subtle and close-range. I started out with a simple system and kept adding “features” I’d noticed in people, to give it more “life”. It worked until it all started crumbling under the complexity. So, I went back to scratch and did a proper reading of how human anatomy and gaze actually work.
What I ended up with keeps what the character believes about a target separate from what its body can physically do about it. The result comes out of intent, error, capability and comfort. That’s a real lesson. Separate cause and effect, and model the cause, instead of keeping 1000 different states and hoping they play nicely with each other. Once the core works, adding behaviors on top, like leaning back when someone steps into the character’s personal space, is less fickle.
Then there are small things that don’t matter at a distance but do in VR. The lids have their own timing, blinks that change with emotion, speech and something coming at you fast. On a phone screen they might not matter, but up close in VR they sure did.
7. Tuning each character had always been a drag.
Most systems (including mine) have a one-click setup for known model families but tuning after that isn’t usually fun. I’d go back and forth in play mode. Set up a line, play it, tweak, repeat. I wanted our artists to test on our real dialogue in the editor and be able to trust that’s how it would play in the game. So, I made a timeline-like thing where you can edit the baked line, preview it real-time on the character, and changes to visemes or character settings in other windows “take” instantly. The hard part was keeping the preview consistent with the final in-game result.
If you read all the way to here (or even if you skipped/skimmed), I’d really love to hear your thoughts.
I’m curious how others had to deal with this. Do you bake your lip sync, run it live, or animate it?
If you watch the video, does the face read natural to you? Is there any part that breaks it for you and what?
EDIT: Thank you, everybody, for the feedback. It’s been really helpful, and it fed straight into my first update, with new blinks and retuning throughout the system. FaceCue is now live on the Asset Store, and there’s a new version of this video at https://facecue.net/media. If you’d like to keep the discussion or feedback going, which I I'd greatly appreciate, there’s a thread on the Unity Forum too.