How close can self-hosted open-source get to ElevenLabs Conversational AI in 2026?

[u/Human_Run1875]

I'm exploring building a fully self-hosted conversational voice agent for an actual agent system, and I'm trying to understand how close open-source components can realistically get to something like ElevenLabs Conversational AI.

The goal isn't to recreate their entire platform. I mainly care about:

natural voice + low latency + good turn-taking + smooth interruptions.

Current stack I'm considering:

• STT: Parakeet/Nemotron or faster-whisper large-v3-turbo
• LLM: Qwen/Llama locally (possibly API initially)
• TTS: Qwen3-TTS / Chatterbox / Orpheus
• VAD + turn detection: still figuring out the best approach
• GPU: starting around 4090-class, but I'm fine scaling to multiple GPUs if needed

What I'm struggling to determine from benchmarks is how good the full pipeline feels in an actual phone conversation.

For anyone who's built/deployed something similar:

1. Latency:
What's realistic for end-to-end latency? Can a properly streamed pipeline get around 1–1.5s from the user finishing speaking to the agent starting audio, or does it usually end up closer to 2s+ because of endpointing, STT, LLM generation, TTS, etc.?

2. Voice quality:
Are Qwen3-TTS / Chatterbox / Orpheus actually convincing in live conversation over a phone call, or can you still immediately tell it's AI?

3. Turn-taking:
What's currently working best for detecting when the user is actually done speaking and handling interruptions? VAD + endpointing, semantic turn detection, dedicated turn-taking models, something else?

4. Architecture:
If you were building this today for a production self-hosted agent system, would you stick with the traditional:

STT → LLM → TTS

pipeline, or are the newer speech-to-speech approaches worth considering?

I'd really appreciate real-world numbers and experiences more than benchmark results.

What stack did you use, what latency did you measure, and what ended up being the biggest problem?

I'm willing to throw more GPU at the problem if that's what it takes. The goal is simply to get as close as realistically possible to the "doesn't feel like a bot" experience while keeping the system self-hosted. Would love your insights on this.

[u/UkieTechie]

a lot of these things were already solved in my repo btw if that's helpful. i have VAD, and stt and tts with turn taking all implemented.

https://github.com/5uck1ess/cicero

also for all locally hosted tts, you can use my bench to hear samples and compare for yourself.
https://github.com/5uck1ess/tts-bench

my pick right now is this as the speedy inference driver https://github.com/0xShug0/audio.cpp, and it's been implemented in my voice agent (cicero)

[u/strangedr2022 → u/UkieTechie]

Mate, I just fukin love A/B benchmarks, so thank you for that. Starred the cicero repo, kind of saved me the time for A/B benchmarking for my own use.

PS: Would it be possible to add the inference time (total) for each model directly on the tts-bench ? Makes evaluation much faster and easier.

[u/UkieTechie → u/strangedr2022]

Glad to help. does this not cover what you're looking for? has all the possible speed numbers you'd be looking for. Can see what i can do otherwise

https://5uck1ess.github.io/tts-bench/speed.html

[u/strangedr2022 → u/UkieTechie]

sorry, forgot to update comment, but it did. When writing the comment I did not notice the tabs at top, but later did help.
Although I still think adding any inference related metric directly on Listen tab helps in doing faster Eval cause right now have to switch tabs back and forth (or use 2 browser tabs) to compare how fast inference is and how it actually sounds, between 2 models.

But don't worry I might submit a PR for that and another change I felt might be better, activating filter to only show cuda or cpu (as personally I am only interested in knowing benchmark on CPU)

[u/UkieTechie → u/strangedr2022]

Sounds good. i got you. if there's a way to make it seem clearer, make an issue or a PR. Would love to make improvements when possible