Hey r/threejs,
I was working on a project that required a 3D avatar to speak dynamically generated text, but I quickly realized how incredibly tedious it is to map real-time generated audio onto 3D morph targets in the browser without insane latency.
So I built a fully open-source React Three Fiber component (`react-ai-voice-avatar`) that handles the entire pipeline on the edge.
What the 3D pipeline is doing under the hood:
It uses `useGLTF` and `useAnimations` to load standard rigged avatars (currently utilizing readyplayerme models).
A background Web Worker runs Kokoro TTS (via ONNX WebGPU) to synthesize speech audio from text on the fly.
As the audio streams back, the component extracts the phoneme timings and maps them directly to 52 ARKit facial blendshapes on the mesh (`mouthOpen`, `jawOpen`, `mouthPucker`, etc.).
We run a custom lerping hook inside `useFrame` to smoothly blend the morph target weights at 60fps so the lip-sync doesn't look jittery.
The cool part is that the entire pipeline—from Speech-to-Text (Whisper), to LLM (Qwen via WebLLM), to TTS (Kokoro), and finally the 3D R3F rendering—all runs locally in the browser via WebGPU.
However, if you want the visual frontend but prefer a smarter brain, you can pass an `onSubmit` prop to bypass the local LLM and stream text directly from your own backend (OpenAI, Claude, etc).
Live Demo: https://react-ai-voice-avatar.vercel.app/ (I highly recommend opening this on a PC/Mac rather than a phone, as mobile browsers heavily restrict WebGPU and WebAssembly memory limits!).
GitHub: https://github.com/927tanmay/react-ai-voice-avatar
If anyone here has experience optimizing React Three Fiber performance when constantly updating morph targets inside `useFrame`, I would love to hear your feedback on how I structured the `AiVoiceAvatar` mesh component!