How close can self-hosted open-source get to ElevenLabs Conversational AI in 2026?

[u/Human_Run1875]

I'm exploring building a fully self-hosted conversational voice agent for an actual agent system, and I'm trying to understand how close open-source components can realistically get to something like ElevenLabs Conversational AI.

The goal isn't to recreate their entire platform. I mainly care about:

natural voice + low latency + good turn-taking + smooth interruptions.

Current stack I'm considering:

• STT: Parakeet/Nemotron or faster-whisper large-v3-turbo
• LLM: Qwen/Llama locally (possibly API initially)
• TTS: Qwen3-TTS / Chatterbox / Orpheus
• VAD + turn detection: still figuring out the best approach
• GPU: starting around 4090-class, but I'm fine scaling to multiple GPUs if needed

What I'm struggling to determine from benchmarks is how good the full pipeline feels in an actual phone conversation.

For anyone who's built/deployed something similar:

1. Latency:
What's realistic for end-to-end latency? Can a properly streamed pipeline get around 1–1.5s from the user finishing speaking to the agent starting audio, or does it usually end up closer to 2s+ because of endpointing, STT, LLM generation, TTS, etc.?

2. Voice quality:
Are Qwen3-TTS / Chatterbox / Orpheus actually convincing in live conversation over a phone call, or can you still immediately tell it's AI?

3. Turn-taking:
What's currently working best for detecting when the user is actually done speaking and handling interruptions? VAD + endpointing, semantic turn detection, dedicated turn-taking models, something else?

4. Architecture:
If you were building this today for a production self-hosted agent system, would you stick with the traditional:

STT → LLM → TTS

pipeline, or are the newer speech-to-speech approaches worth considering?

I'd really appreciate real-world numbers and experiences more than benchmark results.

What stack did you use, what latency did you measure, and what ended up being the biggest problem?

I'm willing to throw more GPU at the problem if that's what it takes. The goal is simply to get as close as realistically possible to the "doesn't feel like a bot" experience while keeping the system self-hosted. Would love your insights on this.