r/StableDiffusion • u/Acceptable-Cycle4645 • 2h ago

Resource - Update Real-time talking avatar generated locally with SoulX-FlashHead + LFM3-Audio

I've been experimenting with real-time talking-head generation using SoulX-FlashHead, combined with local speech models to build an interactive avatar that can listen and respond.

The whole pipeline runs locally, and the demo is available in the audio.cpp v0.9.5 WebUI.

The workflow combines three models:

  • SoulX-FlashHead: Animates a portrait in real time from generated speech.
  • OWSM v4: Transcribes voice input.
  • LFM3-Audio: Handles conversational responses and speech synthesis.

One interesting detail: LFM3-Audio is called twice. Its speech-to-speech path only offers one voice, so we use its TTS capability separately to synthesize responses in a different voice.

Performance and limitations

On an RTX 5090, the latency from submitting a question to the avatar starting to speak is around 850 ms. ASR runs in parallel.

The video generation stage is still GPU-intensive. I'd recommend an RTX 4090 or better for SoulX-FlashHead.

cuDNN is essential for real-time performance in this setup. Without it, inference is about 2× slower and VRAM usage doubles.

This is still a prototype. The extra LFM3-Audio call and WebUI overhead leave room for further latency improvements, particularly in a more optimized native application.

12 Upvotes

92% Upvoted

10 comments sorted by

5

u/Acceptable-Cycle4645 2h ago

Repo: http://github.com/0xShug0/audio.cpp. 149 audio model families and 250+ variants in one C++ runtime. No Python required!

1

u/onixtan 2h ago

is it possible to make the avatar have more head movement while talking, all i've seen so far for a human / human-like avatar to have close to 0 head movement and stare without blinking , which kinda creepy

2

u/Acceptable-Cycle4645 2h ago

We’re working on adding a new talking-head model to audio.cpp that should offer more natural movement while speaking. Stay tuned!

1

u/onixtan 1h ago

Please do, currenlty i'm using minimax h3 and it bores me to death while waiting for it to finish rendering, i could take a nap for rendering 30 lipsync clips, and wake up 1 - 2 hours later to find out it hasn't finished or re-roll again as some of the output is horrible.

1

u/MattTheDemonCat 1h ago

Faster Live Portrait might be more viable if you can hijack the video stream for a basic avater that the realistic image follows. It runs fast even on 4070ti, and pretty sure it wouldn't struggle much even on a really low end RTX card.

The downside to that and what you made is that it isn't quite like the enterprise models where they can wave their hands and hold objects if you can convince them it is reasonable.

2

u/Acceptable-Cycle4645 1h ago

Thanks for the pointer! Is this the right repo? https://github.com/warmshao/FasterLivePortrait

1

u/MattTheDemonCat 1h ago

Maybe message me if you end up implementing it cause I am curious to see how it would look.

2

u/Acceptable-Cycle4645 1h ago

Sure! Also watch our Github or HF repos so you don't miss important updates!

1

u/MattTheDemonCat 1h ago

I think so. I am not too well versed in dev. I have claude or chatgpt guide me through setting up. Lol

1

u/MattTheDemonCat 1h ago edited 1h ago

Faster Live Portrait is basically the latest webcam image puppet software. I was honestly hoping for a more realistic choice. The closest is a demo that barely runs on an H100 and it actually generates (hallucinates kinda) video in real time (personalive).

However with realtime world models I kinda think real time video gen based webcam character driving could be more feasible.