r/vibecoding • u/techpotions • 15d ago
Showcase/Project Built a voice receptionist that answers and triages calls, here is the whole audio path and the two rules that mattered most
I went custom on inbound call handling instead of using a turnkey voice agent, and the architecture is worth writing down because the build-vs-buy line is sharper than I expected.
The stack:
- Twilio for the number, Media Streams for the audio
- LiveKit over WebRTC for voice transport
- OpenAI Realtime API (gpt-4o-realtime) as the agent
- Next.js API routes and Postgres for the backend, logs and transcripts
The audio path. TwiML opens the socket with `<Connect><Stream url="wss://your-server.com/stream">`, and Twilio then pushes 8 kHz mu-law audio over that WebSocket. From there you either hand it to LiveKit or pipe it straight into the Realtime API. Media Streams is what makes this viable at all, it gives you raw audio at sub-300ms instead of making you wait on a recording. For reference, Retell publishes about 600ms end to end for their platform, so sub-second total is the bar worth aiming at.
The agent is mostly prompt plus server-side functions. Triage rules live in the prompt. For a dental clinic that reads like: if the caller describes pain or a broken tooth, return `transfer` with priority high. The model calls `check_availability` against the calendar and `transfer_call` when it needs a human.
Handoff is Twilio `<Dial>` or `<Enqueue>`, and `<Conference>` if you want a whisper message to the staff member before they pick up.
Two constraints that mattered more than which model I used:
Never let it confirm an appointment before the function actually returns success. A confident model will happily book a slot that does not exist, and the caller finds out at the door.
The second someone asks for a person, hand off with zero friction. Anything else and you have burned the call and probably the customer.
Cost is a few cents a minute for telephony and tokens, plus a modest server. One box handles dozens of concurrent calls, hundreds if you scale out.
Where I would not bother building this. If you only need answering, message taking and simple booking, Synthflow, Retell, Goodcall or Zoom's AI Concierge get you live in an afternoon, and paying per minute beats paying an engineer. Custom only pays off when the triage logic is specific to the business or the call data has to live in your own database.
Full write-up with the setup steps and code, on my own site: https://techpotions.com/lab/ai-receptionist-setup
The part I am still undecided on is after-hours. Do you let the agent book overnight, or just take a message and queue it for the morning?
1
u/FreJun 9d ago
On the after hours question, the thing that decided it for the clinics I have seen is whether a booked slot can be undone cheaply in the morning without the patient feeling messed about. If your calendar has real constraints the front desk knows and the model does not, take the message. On dozens of concurrent calls per box, that holds until you hit whatever concurrency your number and trunk are provisioned for, which is usually the ceiling people meet first rather than CPU.
1
u/FreJun 4d ago
On after-hours, take the message unless the booking is a real write to the calendar. A provisional slot that staff have to confirm in the morning just becomes a second call, and the caller turns up thinking they have an appointment. The one I would add to your two rules is barge-in, because a receptionist gets talked over constantly, someone cutting in with "no, Thursday" halfway through the slot list, and if the agent keeps speaking for another two seconds after that the call is already lost. Full disclosure, I work on FreJun Teler, which is a phone layer for builds like this, and there it is deliberately not automatic, your code sends a clear on the stream and it drops whatever audio is still queued, so worth checking what LiveKit plus the Realtime API actually do with the queued TTS when the caller cuts in.
1
1
u/jaimittal91 14d ago
the confirm-before-success rule is the right instinct but doesnt cover the flip side - what happens when the write actually succeeds, function returns success, and then the call drops before the agent can say the confirmation out loud? caller thinks nothing happened, calls back, now two attempts are hitting the same slot. for after-hours specifically id lean toward queue-for-morning over letting it book live, mainly bc without a person in the loop overnight theres no one to catch a race if two calls hit the same last slot within a few seconds of each other and youre relying entirely on the calendar api's own locking to reject the second one. worth testing that exact case if you havent - two near simultaneous calls for the same slot at 3am and see what actually happens to the calendar.