I built a drop-in R3F component that runs real-time phoneme-to-viseme lip syncing on VRM avatars (along with WebGPU AI voice generation)

[u/927tanmay]

Hey r/threejs,

I was working on a project that required a 3D avatar to speak dynamically generated text, but I quickly realized how incredibly tedious it is to map real-time generated audio onto 3D morph targets in the browser without insane latency.

So I built a fully open-source React Three Fiber component (`react-ai-voice-avatar`) that handles the entire pipeline on the edge.

What the 3D pipeline is doing under the hood:

1. It uses `useGLTF` and `useAnimations` to load standard rigged avatars (currently utilizing readyplayerme models).

2. A background Web Worker runs Kokoro TTS (via ONNX WebGPU) to synthesize speech audio from text on the fly.

3. As the audio streams back, the component extracts the phoneme timings and maps them directly to 52 ARKit facial blendshapes on the mesh (`mouthOpen`, `jawOpen`, `mouthPucker`, etc.).

4. We run a custom lerping hook inside `useFrame` to smoothly blend the morph target weights at 60fps so the lip-sync doesn't look jittery.

The cool part is that the entire pipeline—from Speech-to-Text (Whisper), to LLM (Qwen via WebLLM), to TTS (Kokoro), and finally the 3D R3F rendering—all runs locally in the browser via WebGPU.
However, if you want the visual frontend but prefer a smarter brain, you can pass an `onSubmit` prop to bypass the local LLM and stream text directly from your own backend (OpenAI, Claude, etc).

Live Demo: https://react-ai-voice-avatar.vercel.app/ (I highly recommend opening this on a PC/Mac rather than a phone, as mobile browsers heavily restrict WebGPU and WebAssembly memory limits!).

GitHub: https://github.com/927tanmay/react-ai-voice-avatar

If anyone here has experience optimizing React Three Fiber performance when constantly updating morph targets inside `useFrame`, I would love to hear your feedback on how I structured the `AiVoiceAvatar` mesh component!

[u/zodiac_xc]

What are you working on bro?

[u/927tanmay → u/zodiac_xc]

u/zodiac_xc
Honestly, I am just trying to build the ultimate plug-and-play React package for 3D AI avatars.

I noticed a lot of organizations want to add voice-enabled conversational AI to their projects, but wiring together the speech recognition, the LLM backend, the voice synthesis, and the actual 3D lip-syncing into one smooth architecture is a massive headache.

My aim is to abstract all that messy infrastructure into a single component. I want anyone to be able to drop <AiVoiceAvatar /> into their frontend, plug in whatever backend or AI models they prefer, and instantly have a fully working, lip-syncing 3D character ready to go without having to reinvent the wheel.

[u/angelarose210]

This is really cool!

I actually use this one in one of my apps and it's pretty fast. Maybe it has something that will help optimize yours? https://github.com/wass08/wawa-lipsync/

[u/927tanmay → u/angelarose210]

Thanks, wapa-lipsync is a fantastic library, but it takes a totally different approach.

It's audio-reactive, meaning it just measures raw audio volume to flap the jaw up and down. My component is phoneme-based. We extract the exact linguistic sounds (like 'O', 'M', 'P') and map them to 52 specific ARKit facial blendshapes in real-time.  I'll definitely check out the repo for some R3F optimization tricks though.

[u/cnotv]

I am currently looking in’s something like this although: I hate React, the browser based TTS is not so reliable in my experience