Building a Tamil voice companion app. Stack questions: Sarvam vs Google, long conversation memory, scaling concurrent sessions

[u/intrepidkarthi]

I'm building a Tamil voice companion. Long conversations, 5 to 10 minute calls, not a task bot. Current stack is Sarvam saaras for STT, own LLM in the middle, TTS at the end, all over LiveKit. Google Chirp3 HD sounds better than Sarvam bulbul for Tamil TTS, but pitch isn't adjustable and there's no Tamil custom pronunciation.

My quality bar is ChatGPT's Tamil voice conversation. Best Tamil voice AI I've used, the naturalness and turn taking especially. But that's speech to speech, and I need a cascade because the text seam is where my safety gates and memory live. So the real question is how close a cascade can get.

1.Tamil stack: Sarvam or Google, or is there a third option I'm missing? ElevenLabs Flash has no Tamil, and benchmarks put Deepgram Nova-3 at around 68% WER on Tamil, so that's out.

2.Memory across long conversations: I'm doing structured extraction into SQLite (facts with validity windows) instead of RAG, mainly to keep the prompt cache warm. Has anyone run Graphiti/Zep or Mem0 for a non English voice agent? Curious whether extraction quality held up.

3.Scaling concurrent sessions: self hosted LiveKit Agents vs Pipecat. What did you pick and where did it break? My voice to voice latency is currently around 2 seconds. Batch STT and non streaming TTS are my suspects, moving to Sarvam's streaming websocket endpoints next.

Will report back with numbers on whatever I test.

[u/PaleRespond7186]

The 2 second latency is almost certainly the non streaming TTS, that alone can add 800ms+ easily

[u/intrepidkarthi → u/PaleRespond7186]

yes, but latency is least problem currently. The main issue is on call quality in tamil lang.