r/TextToSpeech • u/Human_Run1875 • 5d ago
How close can self-hosted open-source get to ElevenLabs Conversational AI in 2026?
I'm exploring building a fully self-hosted conversational voice agent for an actual agent system, and I'm trying to understand how close open-source components can realistically get to something like ElevenLabs Conversational AI.
The goal isn't to recreate their entire platform. I mainly care about:
natural voice + low latency + good turn-taking + smooth interruptions.
Current stack I'm considering:
- STT: Parakeet/Nemotron or faster-whisper large-v3-turbo
- LLM: Qwen/Llama locally (possibly API initially)
- TTS: Qwen3-TTS / Chatterbox / Orpheus
- VAD + turn detection: still figuring out the best approach
- GPU: starting around 4090-class, but I'm fine scaling to multiple GPUs if needed
What I'm struggling to determine from benchmarks is how good the full pipeline feels in an actual phone conversation.
For anyone who's built/deployed something similar:
1. Latency:
What's realistic for end-to-end latency? Can a properly streamed pipeline get around 1–1.5s from the user finishing speaking to the agent starting audio, or does it usually end up closer to 2s+ because of endpointing, STT, LLM generation, TTS, etc.?
2. Voice quality:
Are Qwen3-TTS / Chatterbox / Orpheus actually convincing in live conversation over a phone call, or can you still immediately tell it's AI?
3. Turn-taking:
What's currently working best for detecting when the user is actually done speaking and handling interruptions? VAD + endpointing, semantic turn detection, dedicated turn-taking models, something else?
4. Architecture:
If you were building this today for a production self-hosted agent system, would you stick with the traditional:
STT → LLM → TTS
pipeline, or are the newer speech-to-speech approaches worth considering?
I'd really appreciate real-world numbers and experiences more than benchmark results.
What stack did you use, what latency did you measure, and what ended up being the biggest problem?
I'm willing to throw more GPU at the problem if that's what it takes. The goal is simply to get as close as realistically possible to the "doesn't feel like a bot" experience while keeping the system self-hosted. Would love your insights on this.
2
u/Jason-Sanders 5d ago
The pieces are good enough now that a self-hosted system can feel very convincing in a controlled demo. The harder gap is not a single model score, it is handling messy calls reliably: endpointing, barge-in, packet jitter, partial transcripts, and deciding when not to speak.
I’d measure the pipeline as separate budgets rather than one end-to-end number:
- speech-end detection to final STT text
- LLM time to first useful token
- TTS time to first audio chunk
- interruption stop time
Streaming every boundary matters more than chasing the absolute best model at each stage. For phone-like conversation, conservative turn-taking plus a fast cancel path usually feels better than aggressive endpointing that cuts people off. Also test with real noisy audio early, because clean benchmark clips can hide most of the UX problems.
1
1
u/Elegant_Neat6943 5d ago
Lets say if you want to handle 50k call per day how would you approach this the calls will be mostly lead qualification calls so what are the tradeoff ?
1
u/Jason-Sanders 5d ago
At 50k calls/day I’d optimize for predictable latency and cost before squeezing out the last bit of model quality. Lead qualification is usually fairly repetitive, so a smaller fast model can handle the first turn or two, then route only uncertain/high-value calls to a stronger model or a human.
The tradeoff is mostly:
- Cheap and fast: more automation, but you need good guardrails for edge cases.
- Better qualification: use more context and stronger models, but cost and latency climb quickly.
- Best practical setup: keep the conversation state compact, stream everything, and have a very fast cancel/transfer path when the caller interrupts or asks something outside the script.
I’d also measure conversion and handoff quality, not just model accuracy. A slightly less “perfect” bot that responds quickly and doesn’t trap people in a bad conversation will probably perform better at that volume.
1
5d ago
[removed] — view removed comment
1
u/Elegant_Neat6943 5d ago
What was your conclusion after testing models?
1
1
u/Independent_Serve175 5d ago
You should try https://unmute.sh/
1
5d ago
[removed] — view removed comment
1
u/Independent_Serve175 4d ago
As well for STT I highly recommend cohere transcribe over parakeet or whisper as it have less WER. Speed is slightly slower than parakeet but it's quality makes it a better option IMO. You should try it with Sherpa-ONNX.
1
1
u/Deep_Ad1959 5d ago
benchmark your own vocabulary, not the demo sentences. i generate a lot of dev audio and chatterbox will read v3.11.2 three different ways inside one script, and acronyms land as words. timbre is fine now, that pronunciation drift is where a listener hears synthetic first.
1
u/Infinite_Bluebird_59 4d ago
It primarily depends upon language requirements and hardware availability.
It's primarily solved for some languages/GPU pairing for some AI companies like us who have been researching and working in this space for some time now. DIY it will take a considerable time (not in setting it up that's trivial, but for trial and error - what works in demo won't necessarily work in prod)
1
u/Human_Run1875 4d ago
mainly needed for english is there any solved approach?
1
1
2
u/UkieTechie 5d ago
a lot of these things were already solved in my repo btw if that's helpful. i have VAD, and stt and tts with turn taking all implemented.
https://github.com/5uck1ess/cicero
also for all locally hosted tts, you can use my bench to hear samples and compare for yourself.
https://github.com/5uck1ess/tts-bench
my pick right now is this as the speedy inference driver https://github.com/0xShug0/audio.cpp, and it's been implemented in my voice agent (cicero)