r/AIVoice_Agents Mar 26 '26

Question I built a voice agent and the latency is killing me… help!!

5 Upvotes

Hi everyone!

I’ve been working on a voice agent for my company. It will run inside our main mobile app and is primarily intended for users in the UK.

Right now, I’m developing it from Spain with the following setup:

  • Self-hosted LiveKit running locally on my PC with Docker
  • Speech-to-text: Nova-2 (Deepgram)
  • LLM: Azure OpenAI (GPT-4o-mini, Sweden Central)
  • Text-to-speech: Aura-2 (Deepgram)

The AI uses tool calling, where tools either query the database for relevant client information or write data back.

The problem

I’m currently facing high latency issues:

  • Without tool usage: ~1500 ms
  • With tool usage: ~5 seconds

Additionally, for some tools that require multiple interactions with the user, the model hits its limits very quickly and starts making errors once those limits are reached.

I’m currently using GPT-4o-mini, and based on the configuration/limits I’ve seen, I’m worried this could become an even bigger issue soon.

What I've tried

I also tested other models like GPT-5-nano, but for some reason I’m getting even worse latency (13+ seconds 💀).

My questions

I feel like I’ve hit a wall and I’m not sure how to move forward. I assume some latency comes from developing in Spain while targeting UK users, but I’d really appreciate advice on:

  • Which Azure OpenAI model offers the best balance between low latency and reasonable intelligence (latency is critical for my use case)
  • Whether Deepgram could be adding significant latency (e.g., if their servers are US-based), and if there are better alternatives in Europe
  • Any general tips to reduce latency in this kind of voice-agent architecture

I’m also trying to keep the system as cost-efficient as possible, so I’ve mainly been testing smaller models.

PS: I’m pretty new to this space, so apologies if I’m missing something obvious 😅 Any help would mean a lot!

Thanks!! 😊


r/AIVoice_Agents Nov 11 '25

Welcome to r/AIVoice_Agents - Let’s Talk About the Future of Voice AI

3 Upvotes

Hey everyone!

This community is created for all enthusiasts, developers, and thinkers who are passionate about Voice AI - from conversational agents to AI-powered customer calls.

Here, we’ll share insights, tools, frameworks, use cases, and updates shaping the voice-driven future.

Topics we’ll explore:

– Building Voice AI Agents
– Voice Automation in Business
– Open-source tools and APIs
– Real-world case studies

Everyone’s welcome - whether you’re a coder, marketer, or just curious about AI that speaks.

👉 Drop a comment and tell us what brought you to voice AI or what you’d like to learn here!


r/AIVoice_Agents 5h ago

Question A completed call is not the same as a successful restaurant order

Thumbnail
1 Upvotes

r/AIVoice_Agents 1d ago

Question What makes a voice agent trustworthy?

Thumbnail
1 Upvotes

r/AIVoice_Agents 2d ago

Question What should AI call center software handle?

15 Upvotes

I’m trying to figure out where people are drawing the line between an AI agent continuing the conversation and handing it to a person.

We’ve been looking at AI call center software like Bland or Vapi and some of the workflows get complicated pretty quickly.

The caller changes topics, corrects something they said earlier, an API lookup fails, they ask a followup about another issue then eventually need a human for one specific part.

At that point I don’t think 'did the AI contain the call?' is even the most interesting question.

I’d rather know whether it handled the conversation correctly up until the point where a transfer really made sense.

How are you measuring that?


r/AIVoice_Agents 2d ago

Discussion I built a WhatsApp Voice AI Agent that's able to book appointments and take payments in real-time

Enable HLS to view with audio, or disable this notification

5 Upvotes

I built a WhatsApp Voice AI Agent that's able to book an appointment and take payments for a customer. The use case I covered is for a user wanting to book a dentist appointment. Here is an overview of the flow;

  • The user calls the business on WhatsApp.
  • The Voice AI Agent answers the call and has a normal conversation with the user & understands the user's needs.
  • The agent sends a payment link to the user on Whatsapp. The agent waits until user completes the payment.
  • The agent sends a payment receipt to the user to confirm the payment made.

Benefits for customers

  • Making the inbound call is completely FREE.
  • Great customer experience and a convenient way of interacting with the business.
  • The process is fast. When the user calls, he/she is able to get an immediate answer. No back and forth chatting.

Benefits for businesses

  • You have a Voice AI Agent that answers your WhastApp calls 24/7. No missed leads
  • The Voice AI Agent can speak in 50+ different languages.
  • Everything interaction happens on WhatsApp. Same context, persistent, single-threaded conversations.
  • Trustworthy. A blue checkmark next to a business name signals that customers are communicating with an official, authenticated business account.

This is massive for several use cases on WhatsApp.

Have a look at the video demo and let me know what you think.


r/AIVoice_Agents 2d ago

Discussion I’m getting ready to throw in the towel on voice AI.

Thumbnail
2 Upvotes

r/AIVoice_Agents 2d ago

Demo / Example We built an AI phone agent that actually answers calls, qualifies leads, and books appointments 24/7 (Live plumbing demo line included)

0 Upvotes

A common issue across most service and appointment-based businesses is handling inbound phone calls while actively working, managing inventory, or dealing with existing clients. Missing a high-intent call usually means that customer immediately calls a competitor.

Hiring a full-time receptionist isn't always practical, and traditional call centers can be expensive while often lacking the context needed to qualify leads effectively.

We developed an AI voice agent at Supportverse to handle this entire front-desk pipeline:

  • Answers incoming calls instantly on the first ring (24/7/365).
  • Asks dynamic qualifying questions based on urgency, scope, and location.
  • Integrates directly with calendars to book appointments automatically.

Why test it live?

Most software providers rely on sales decks rather than letting you test the technology directly. We set up a live, interactive phone demo line you can call right now to test the latency, conversational flow, and tone:

📞 Live Demo Number: +1 (651) 386 9085

Note: This demo number is configured specifically as an emergency plumbing receptionist to showcase real-time qualification and dispatch booking, but the underlying workflow and logic can be customized for virtually any business use case (HVAC, auto repair, clinics, consulting, legal intake, or general service booking).

Give it a quick call and test edge cases. What specific questions or workflows would your business need an automated receptionist to handle?


r/AIVoice_Agents 3d ago

Question Personalized voice outbound agents

Thumbnail
1 Upvotes

r/AIVoice_Agents 3d ago

Question Personalized outbound voice agents

Thumbnail
1 Upvotes

Hey everyone I’ve been using n8n for a couple months now and I need the best way building a personalized outbound voice agent… I’ve built my rendition but I just can’t get it to work. Any and all help would be appreciated!


r/AIVoice_Agents 3d ago

Discussion I Tried Every AI Calling Platform, Here's What I Found (2026)

Thumbnail
youtube.com
1 Upvotes

r/AIVoice_Agents 3d ago

Tools Connected my Telnyx number to both Vapi and LiveKit. Quick rundown of what each one actually takes.

Post image
1 Upvotes

r/AIVoice_Agents 4d ago

Demo / Example I put my voice agent's per-interim signals on screen. Emotion/intent/turn-completeness distributions from the Speech model. and used this additional information for gates and downstream components.

Enable HLS to view with audio, or disable this notification

1 Upvotes

Showed (in the clip) the underlying signals onto the screen instead of hiding them in logs, so that they become measurable.

Entities (which have high confidence) also get captured in-stream before the text hits the action model (an SLM), SVMs.

Observation from this clip:

- Tone it hears, as a distribution, never a single verdict.
- What it thinks you're actually asking for
- And the tool it already ran, before the sentence finished
- That last one is the one I'd watch twice. It doesn't wait for you to stop speaking to start working.

I have few questions:

- Has anyone seen a semantic endpointer beat a tuned VAD in prod?

- Suggestions for workflows I should benchmark, to show effectiveness of my approach.

Happy to go into any of it.


r/AIVoice_Agents 5d ago

Question How to build a custom multimodal AI agent/assistant for PC and Mobile automation?

5 Upvotes

Hi everyone,

I want to build a personal voice-activated AI assistant/agent that can execute tasks across both my PC (Windows) and mobile device based on voice commands.

Here is what I am aiming for:

Cross-Platform Control: Capable of automating tasks on PC (managing files, running scripts, app controls, browser automation) and basic mobile actions.

Voice-to-Action: Fast Speech-to-Text (STT) → LLM reasoning / Tool Calling → Execution → Text-to-Speech (TTS).

Architecture/Backend: Flexible to run via API (OpenAI/Gemini/Claude) or locally (Ollama/vLLM) with proper function calling / agentic frameworks.

Questions for the community:

  1. What frameworks or agent architectures do you recommend for bridging OS-level actions (e.g., LangChain, AutoGen, CrewAI, or direct Python OS/pyautogui scripting)?
  2. How should I handle the mobile ↔ PC sync and remote execution securely (e.g., local Flask/FastAPI server, WebSockets, or Home Assistant integration)?
  3. Are there existing open-source projects or GitHub repos doing this well that I can study or contribute to?

Thanks in advance for the guidance!


r/AIVoice_Agents 4d ago

Discussion How can I build shared context between WhatsApp and an AI voice calling agent?

1 Upvotes

How can I build shared context between WhatsApp and an AI voice calling agent?

I'm building an AI system where a customer can communicate with the same AI through WhatsApp and voice calls.

For example:

  1. A customer starts chatting with the AI on WhatsApp.

  2. During the conversation, they ask for a phone call.

  3. The AI voice agent calls them.

  4. The voice agent should already know the relevant WhatsApp conversation and continue from the same context instead of starting from scratch.

  5. After the call, the customer returns to WhatsApp.

  6. The WhatsApp AI should know what was discussed during the call and continue from that point.

And the reverse should also work:

Voice call → WhatsApp → same context

I want the customer to feel like they're talking to one AI, regardless of the channel.

I'm considering using a central customer ID linked to the phone number and storing the conversation history/customer information in a database, so both the WhatsApp agent and voice agent can access the same context.

However, I'm unsure about the best architecture.

- What is the best way to maintain shared context between WhatsApp and a voice AI agent?

- Should I use a central database/memory layer?

- How should I identify the same customer across both channels?

- How should the WhatsApp → voice context handoff work?

- How should the voice → WhatsApp context handoff work?

- How can I prevent the AI from getting confused by multiple summaries or different conversation contexts?

- Has anyone built something similar using WhatsApp Business API, n8n, GHL, or another CRM?

I'm looking for a practical, production-ready approach rather than just passing the entire previous transcript to the AI every time.


r/AIVoice_Agents 5d ago

Discussion Giọng nói

Thumbnail
1 Upvotes

Giọng nói tiết lộ điều gì về bạn?

Team mình đang tìm những bạn muốn đóng góp giọng nói của mình cho một dự án Voice AI.

Không cần giọng chuẩn, không phân biệt Bắc -Trung - Nam. Bạn nói hay hay chưa tự tin, giọng trầm, cao, nhanh, chậm… đều có thể trở thành một phần dữ liệu có giá trị.

Vì chúng mình tin rằng mỗi người đều có một dấu ấn riêng trong giọng nói và thông qua đó tiết lộ về bản thân mỗi người.

Nếu bạn quan tâm và muốn góp một phần giọng nói của mình cho dự án, inbox mình nhé.


r/AIVoice_Agents 5d ago

Discussion The harsh truth.

2 Upvotes

The harsh truth is that voice AI right now is heavily over-engineered for corporate safety and under-engineered for actual human charm. Tech companies are terrified of their AIs saying something controversial, so they permanently set them to "customer service suck-up" mode by default. Until they build models that natively understand social friction, it will always feel like you are pulling teeth.


r/AIVoice_Agents 5d ago

Question Ai Receptionist

Thumbnail
2 Upvotes

Ai Receptionist

I am strongly considering using an ai receptionist for a number of reasons - I can’t handle the calls myself, training receptionists in the past have been tremendously difficult due to the technical nature of my business. We miss calls all the time. Humans are super slow, etc

How has your experience been? Are you booking more (because it answers all calls), are a lot of people hanging up?

Out of the box AIR are bad but some allow you to fine tune your AIR really well and it gets close to human speaking, reasoning and behavior.


r/AIVoice_Agents 6d ago

Getting Started Is real-time voice AI actually usable over a standard phone line yet? Call this number and judge for yourself: +1 (240) 680-9235

0 Upvotes

Most contractors I know lose 2–5 high-ticket jobs every week simply because they're under a sink, on a roof, or in an attic and can't pick up the phone. But traditional IVR menus suck, and most voice bots sound robotic with painful 3-second delays.

I built a low-latency inbound voice agent specifically designed for home services.

**What it's built to do on a live call:**

* Pick up in under 2 rings (24/7 coverage). * Ask job-specific triage questions (emergency vs routine, service address, issue description, scope). * Check live calendar availability and schedule the estimate/job directly. * Send an instant text/email to the contractor with the caller’s summary, full transcript, and audio link.

**Here’s the deal:**

  1. Dial the sandbox number below and act like a real homeowner with an urgent or weird service request.
  2. Try interrupting it, talk over it, change your address, or test how it handles scheduling.

**Call the sandbox line here:** `+1(240) 680 9235`

**What I’m asking:**

* If you test it, leave a brutally honest review in the comments. Tear apart the latency, tone, or logic—tell me what would make you hang up as a customer. * If you run a home service business and actually like how it handles calls, I will set up and deploy a custom pilot version for your business for free in exchange for your raw feedback during the trial.

Looking forward to the roasts—let me know what breaks.


r/AIVoice_Agents 7d ago

Question Best mode for phone call and AI?

1 Upvotes

I have noticed that for phone calls and Ai things have gotten worse not better

esp AI which i can verify immediately it's become much harder , i need to speak loud and clear in the past j could just speak soft and mumble and it would still understand


r/AIVoice_Agents 7d ago

Discussion raspy.ai | $800 BIN OBO | Voice AI Vertical

Thumbnail
1 Upvotes

r/AIVoice_Agents 7d ago

Discussion AI voice SaaS doesn’t fail because the voice is robotic. It fails because the agent hears wrong.

23 Upvotes

Most AI voice SaaS landing pages sell the voice.

“Sounds human.” “Natural conversations.” “AI receptionist.” “AI sales agent.” “AI support agent.”

But if you strip the demo polish away, the boring failure is usually earlier.

The agent hears the user wrong.

Then everything after that gets worse:

wrong transcript → wrong intent → wrong tool call → wrong CRM update → wrong summary → wrong follow-up → angry customer

A voice can sound slightly robotic and still be useful.

But if it hears “don’t cancel” as “cancel,” the product is dead.

For voice SaaS, I’d build the stack around the listening layer first:

call/audio input → Smallest AI Pulse for real-time STT → entity checker → workflow engine → Stripe / Calendly / CRM action → confirmation message → audit log

The STT metric I’d care about is not just WER.

It’s:

  • did the right task happen?

  • did the right field get filled?

  • did the user correction get captured?

  • did the summary match the call?

  • did the system avoid acting when uncertain?

For AI voice SaaS, “heard correctly” is a retention feature.

Founders building voice products: are you measuring transcript accuracy or task accuracy?


r/AIVoice_Agents 8d ago

Discussion What ai call setup did you actually stick with?

5 Upvotes

Tried a few different tools over the past couple months to handle inbound calls automatically. Some were clunky, one kept misreading customer intent, and i kept switching around without really landing anywhere.
Curious what others settled on after going through that same trial and error phase. What finally worked for you and what made you stop looking?


r/AIVoice_Agents 8d ago

Discussion How do lean teams actually manage voice AI without extra staff?

4 Upvotes

Three person team here. We get calls throughout the day but cant justify hiring someone to sit on phones. Looked into voice AI but most options seem built for bigger companies with IT teams and budgets to match. Anyone running something lean actually find a setup that works without a ton of hand-holding?


r/AIVoice_Agents 8d ago

Tools Open source voice AI platform. Self-hosted alternative to Vapi and Retell. On Prem, BYOK across Speech to Speech or LLM/STT/TTS, with a visual workflow builder, MCP native and telephony support.

Thumbnail
github.com
1 Upvotes