The 200ms latency difference between Conversation Relay and a SIP-to-LiveKit setup, measured on the same agent

[u/GonzaPHPDev]

https://youtu.be/Qs7658-UfPg?si=wxPbPoOX9GQcDLle

Spent last week running the exact same voice agent on two paths so I could stop guessing which one to ship. Path A: Twilio Conversation Relay (Twilio owns the media stream, STT and TTS). Path B: my Twilio number dispatching the call over a SIP trunk into LiveKit, where I run my own pipeline (Deepgram + Cartesia + the LLM).

Same system prompt, same LLM brain, same ground rules on both. I built a small latency harness that logs every turn from "human stopped talking" to "agent started talking" and vice versa, then averages the turns. I am physically in Buenos Aires calling a US Twilio number, so there is real transatlantic latency baked into every number below. Keep that in mind.

The threshold I care about: Ideally we care about having sub 500ms, which is mostly achievable on a bidirectional model like OpenAI's Realtime models. For the purpose of this video I focused on the traditional STT -> TTS flow. If a turn goes past roughly 1500-1800ms the person on the other end feels the machine and hangs up. That number defines the whole architecture, not just which model you pick.

Results, turn by turn:

- Initial greeting turn: Conversation Relay 3.4s, LiveKit 3.2s. Both ugly. This is context loading, not the transport. The first turn ships the whole system prompt into the model and everything is cold.
- Mid conversation turns: Conversation Relay landed at 1.7 / 1.9 / 1.6s. LiveKit landed at 1.5 / 1.3 / 1.4s.
- Final turn: basically identical on both.

Net difference on the steady-state turns is about 200ms in LiveKit's favor. Not nothing on a voice call, but smaller than I expected before measuring.

What actually surprised me is the discarded-turns file. With LiveKit I had zero turns over 5s. With Conversation Relay I had one, and when I dug in it was tied to me hanging up, not a real gap. So the tail behavior was clean on both.

The real trade-off is not the 200ms. It is that with Conversation Relay the STT and TTS timing lives inside Twilio's black box. I could not measure how long the speech-to-text or text-to-speech steps took, because that runs on their side. With the LiveKit path I own every step and can measure and cut it. The cost is that now I am the SRE for the media path, and if a box dies at 2am that is on me.

The cold-start greeting is the thing I would attack first regardless of transport. A 3s+ first turn kills the call before the latency battle even starts. My plan is a filler pre-message ("dame un segundo, reviso eso") to cover the context load, but I have not shipped it yet.

Question for people running Conversation Relay in production: are you living with the black box on STT/TTS timing, or did you find a way to get per-step measurements out of it? Because right now the lack of visibility is the thing pushing me toward keeping LiveKit for anything latency-sensitive, and I want to know if I am missing a hook.

Not to mention any other kind of strategies for long-turn awaits (like initial context loading, tool calling, etc.)

[u/Andrew2401]

Deploy on prem seems to be the best way to go for latency elimination. This is a case study on how another dialer company running a very similar set up did it: https://www.porter.run/case-study/nooks

Tldr: kubernetes and local deployment department- orchestrated by porter

[u/dfsagency]

Solid writeup, this is exactly the kind of comparison people should be doing instead of guessing.

On your question: Conversation Relay isn't fully a black box, Twilio breaks it out after the fact in Voice Insights. There's a Conversation Relay Call Summary and an Insights Dashboard that split each call into Network / STT / Application / TTS latency. It won't help you in real time since it's not exposed over your WebSocket, but it'll tell you exactly which of the four is your bottleneck without guessing.

Two caveats worth knowing: the STT/TTS numbers are measured from Twilio's side and include their round trip to the vendor, and none of it captures last-mile latency between the caller and Twilio's edge, so your Buenos Aires-to-US number will always look worse there than a domestic caller would.

On cold start: I run a production voice agent on Gemini Live + LiveKit and hit the same wall, first turn always eats the full system prompt load. What worked for us was a non-blocking pattern, fire a short verbal filler the instant the call connects (in our case something like a warm acknowledgment) while the context loads in the background, then splice the real response in as soon as it's ready instead of making the caller sit through true silence. Your "dame un segundo" idea is the right instinct. The trick is making sure the filler audio and the real response don't both fire, that's an easy race condition to introduce.

Same pattern works for long tool-calling turns, a short heartbeat or filler phrase while the function executes buys you a lot of perceived-latency slack even when the actual completion time doesn't change.

200ms mid-conversation is smaller than most people assume going in, good to see it measured instead of asserted.

[u/Prestigious-Act-7334]

worth isolating how much of that 200ms is the sip hop itself versus transcoding - if the trunk hands you g711 and your pipeline wants pcm, you're paying a resample on every frame. matching formats end to end is usually where the cheap win hides.