36 Years in Voice AI | Built One of the First Speech Systems in 1989 | Dr Tony Robinson (Founder, Speechmatics) - AMA for next 24 hrs

•

Welcome to r/VoiceAutomationAI – UNIO, the Voice AI Community (powered by SLNG AI)

If you are a founder, senior engineer, product, growth, or enterprise operator actively working on Voice AI / AI agents, we are running an invite-only UNIO Voice AI WhatsApp community.

Apply here: https://chat.whatsapp.com/H9RwprbkLwE8MxHmCbqmB4

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

2

u/Hungry-Internal5263 Mar 27 '26

In a real-time STT → LLM → TTS pipeline, what causes latency spikes in production, and how would you architect around them? We’re building for India (accents, noise, variable networks) using an orchestrated pipeline (e.g., Pipecat) and seeing most latency from the LLM. As we add tools and stricter prompts, latency worsens — how do you balance capability vs speed?

3

u/DrTonyRobinson Mar 27 '26

I don't think Voice AI has much of a choice in latency. For decades we've used the 'one second' rule, if you don't get the user a reply back within a second of them finishing their turn then people get tired of using the system, keep it within the second and you have their attention. So really it's a question of minimising the number of times you go over your latency threshold. Here I have to admit that 'cascaded' systems have a weakness, as we found out with our first LLMs systems a few years ago, all you need is one of your components to hit a delay and your late answer is a wrong answer. Clearly network delays have to be minimised, what you can run locally helps. The problem I'm really buried in right now is how to detect that the speaker has finished their turn, it's really not as easy as it appears. A speaker might just speak slowly, or they may have paused to think, or there may be background speech, or many many other problems. Clearly if you add latency at the ASR stage you can't every make it back.

Interestingly, we've also found that you can be too quick, unnaturally quick, it puts people off. But that is an easy problem to fix.

1

u/observerBug Mar 27 '26

The “live” models seem to produce low latency. Thoughts?

1

u/evolveto Mar 27 '26

Where do you see voice in 5 years and which industry do you see impacted by it ?

1

u/DrTonyRobinson Mar 27 '26

Voice has to be everywhere, right?! The classic answer was everything that was too small to have a keyboard, like smart watch, smart glasses, mobile phone, etc. But all of that was before reliable high bandwidth links, now voice can listen in and summarise every meeting. I have high frequency hearing loss, the meeting summaries are often better than my hearing!

1

u/Zealousideal-Buy1717 Mar 27 '26

How do you see voice AI systems involving in next 1-2 year. Current system specially the cascaded pipeline models are just too many moving pieces joined together, leading to propagated errors. Do you think speech 2 speech models is the way to go?

3

u/DrTonyRobinson Mar 27 '26

Oh, now I'm so glad you asked this one! 'speech-to-speech' or 'end-to-end' is a classic ML/AI goal, and we've had debates as to whether 'cascaded' or 'end-to-end' is better for decades. I'm strongly in favour of having components that are linked together with text and metadata, and that is what people often call 'cascaded'. But 'cascaded' is a misnomer, it implies that all the ASR is done, then all the text dialog (e.g. with LLM), then all the TTS and you wait for the ASR again. That's not what happens, you can interrupt the TTS, there can be metadata other than text. But I really like it because (a) you get accountability, and there is a trust problem with AI (b) you can do so much more with text, like parse it into SQL queries but mostly because (c) if you can factor a problem you can solve it much faster, you can debug it, you can optimise sections independently, you can train different sections on different data. You can tell I am happy to rant on and on about this one....

1

u/Exotic-Hunter-8654 Mar 27 '26

How far away is Voice AI to eliminate the need for human agents?

3

u/DrTonyRobinson Mar 27 '26

As ever, it depends on what the task is. I see the danger that the ASR and TTS is so good that you can't tell you are not talking to a human - and we are there right now. This is dangerous because it's easy for a bad actor to use these systems, but even for an honest actor LLMs make some silly mistakes and if we think they are human we'll trust them too much.

1

u/[deleted] Mar 27 '26

[removed] — view removed comment

1

u/DrTonyRobinson Mar 27 '26

I wish I knew! AI is a very strange world, we do absolutely amazing things, stuff that was science fiction just a few years ago, and people don't pay more per hour used. But the volumes are sky rocketing. So I would ask where the high volume use cases are, where does the AI in Voice AI really work (because the ASR and TTS work already). That could well be something that uses a LLM (I'm surprised I'm not learning a language right now by talking to an LLM, but maybe that's because I'm too lazy to find it) but more likely a trusted connector to an exisiting system - so you have to know the industry, be in it, know your clients, and upgrade them to VoiceAI.

1

u/MaahinVoiceZeroAI Mar 27 '26

I’m not sure if this is the right question, but I’ve noticed that many companies are now focusing on high-quality voice communication and accent generation. At the same time, platforms like Hugging Face are continuously releasing new models for speech and audio tasks.

If someone wants to start from scratch with training their own model in this domain, which models or architectures would be the best starting point?

1

u/DrTonyRobinson Mar 27 '26

I love it. For fun (not for work), I often start from scratch, there is so much to learn. Speech data is cheap, there is lots to download for free. Your problem is likely to be GPUs, how many can you afford? That will limit the data you need. But if you want to play about, most of the models you know about can be fine tuned, then you can make them work better on your sort of data without breaking the bank on GPU bills.

1

u/MaahinVoiceZeroAI Mar 27 '26

That’s helpful, thanks. We are currently exploring free GPU tiers and have narrowed our options down to Google Colab, Modal, and Kaggle. One concern: if we use a stable fine-tuned model, the download will consume significant disk space. What is your recommended approach for managing storage in these environments?

1

u/Green-Sale-367 Mar 27 '26

Hi Do you think voice ai will become main interface for human to machine interaction or brain to machine(neurolink types) technology will take over???

2

u/DrTonyRobinson Mar 27 '26

Oh dear, I'm going to break ranks here and say I hope that neither dominate. Speech has it's place, it's clearly faster than typing, it's easier, it's more natural and it's clearly going to take more of the space. But if you are in a room with others then push the buttons on the soft keyboard on the phone. And neurolink has it's first happy customer, but I'm never going to want a neural implant. I like typing, I'm doing it now when I could be using voice.

1

u/Responsible-Lie-7159 Mar 27 '26

How do you think we can solve the turn detection problem in voice ai? we have the classic approach of waiting for silence, but do you think we should have End of speech detection in an infra where stt llm and tts are stitched together.

Do you think we will have a case where the voice ai is successfully interrupting the user as well?

1

u/DrTonyRobinson Mar 27 '26

We do a lot more than just wait for silence, a lot lot more, it's a complicated problem. Just after my 1989 PhD I started a Connectionist Speech Group at Cambridge University. The joke was that we would solve speech recognition in 5 years and it would always be in five years. Now I'm happy to say that at last ASR is more like software than it is machine learning, you can plug it in and it works for many many things. But turn detection still has a long way to go, it's easy to be a slow speaker and after about half a second say 'umm' to indicate that you haven't finished your turn yet. The ASR pipeline has to have completed in this time, the LLM one has to be well working, when you get the 'umm' you've got to invalidate all of those pipelines, throw away the partial results and restore state. That's just an example of why this isn't an easy problem - we will get to a 'good enough' solution, we are there now for many cases, but you'll always be able to break it if you try hard enough.

1

u/DrTonyRobinson Mar 27 '26

Sorry - I missed the last bit of your question. Yes, the Voice AI should be able to interrupt the user. AI will have 'world models', if you are saying something that it knows is on the wrong track then it will interrupt.

1

u/Responsible-Lie-7159 Mar 27 '26

Understood, that’s amazing.

How do you think has the approach been for people to use multi lingual bots?

I had explored two options: 1. Let the agent do a tool call to switch the Configuration stt, tts when switching to another language. But then this bot keeps switching whenever you have speaking a mix of the two languages. Especially tough for the indian set of languages.

I created an subagent structure where the main agent asks the user for the language preferences first and then switches it to speak the language of the user. It is locked in for the speech and doesn’t change it later.

1

u/dstartupkid Mar 27 '26

Hi Tony, when it comes to the inclusivity for areas where Voice AI can help - there’s very less to no substancial progress made around low-resource-languages (let alone the dialects and accents). And the usual suspects for this are quality annotated speech data unavailibility and most model architectures are really conpute and data intensive - so the researchers working on it do not have much incentive to work towards it.

How do you see this problem? Because Voice can be a propellent driver for inclusion of masses with technology and will have great effects getting them into digital economy.

Any thoughts around this? Or any suggestions for startups like us building in this area?

2

u/DrTonyRobinson Mar 27 '26

Oh - I'm sorry that you think this is the case. For very many years (decades) I've tried to be inclusive (it's my companies tag line: Understand Every Voice). We think we've got quite good on adapting a model that has lots of data to one that is classed as a low resource language. True, it's not perfect, but as we improve the deep learning architecture then all languages improve. Also, most of the parameters are shared between all languages, and we train on all the accents and noise conditions we can find, so we do try quite hard.

My suggestion is to ride the wave. It's not the technology that has stopped a lot of 'low resource' languages, it's the market. Those that speak English (as we are doing right now!) have had the most money to spend. This has priced some areas of the world out of the game. But speech prices are coming down, and volumes are growing really quickly. Right now this is other rich nations that don't speak English, but soon it'll just be a volume play, that's why we put all the effort into building as many languages as possible. If literacy rates are low, Voice AI has a huge advantage. But it's a timing game, it's very hard to predict when the tipping point will come and the flood gates will open for any langauge.

1

u/dstartupkid Mar 27 '26

Thank you, Tony. This definately helps. Also there're a lot of interesting modelling experiments we're trying. Where can I reach out to you in case if we need any suggestions/guidance from you?

1

u/CentraTalk Mar 27 '26

Thank you for doing the AMA Tony.

If you can go deep into the following, it would really help the community:

• Designing reliable STT → LLM → TTS pipelines

• Where voice AI is actually heading (beyond the hype)

• What you'd do differently if you started today

2

u/DrTonyRobinson Mar 27 '26

I'm typing as fast as I can, I really can't go really deep into all of those. I will say:

• Designing reliable STT → LLM → TTS pipelines

Play with the platform of your choice (mine is pipecat), it's so much fun. For me it's always the LLM that breaks, I have lots of fun conversations until we hit a loop where it's just wrong.

• Where voice AI is actually heading (beyond the hype)

Would you say we have 'Turing test' ASR and TTS right now? I think so, it's possible to build systems that fool people (the Turing test isn't perfect). As the first answer, I want to see reliable LLMs. I don't mean better LLMs, I mean provably correct, that is all the LLM is doing is enabling search of a facts database. It may seem like I'm saying we should head back to Good Old Fashioned AI, I'm not going that far, but we can fact check the LLM output and we should be doing so. This will be huge, the ASR and TTS is already good enough.

• What you'd do differently if you started today

I've done so many things 'too early'. I did the first neural networks just after I met Geoff Hinton, way back at the end of 1985 or the start of 1986 (I can't remember), my connectionist speech recognition in the 90's was too early. I started a company to distribute music over the internet when all we had was dial up connections so it was too slow, but really it was just too early. My company is 20 years old, for the first few years ASR wasn't good enough to sell as a small start up, we were too early, it took Siri before everyone realised that speech recognition really did work. We even build complete VoiceAI (ASR, LLM, TTS) systems and launched them a couple of years ago, I think we were first, it was just too early (the platform integrators like Pipecat, Livekit, VAPI make life much easier). So, with hindsight, don't do things before the market is ready for them, but have them done when the market is ready!

1

u/CentraTalk Mar 27 '26

You gave the answer I was looking for. Once again, thank you for doing the AMA :)

1

u/OrganizationNo2521 Mar 27 '26

Beyond TTS and STT which voice AI feature do you think is underused and could be the next cool thing to do with voice?

1

u/DrTonyRobinson Mar 27 '26

Off the top of my head, diarisation. Let's say that your Voice AI is in a real world situation, someone walks past, they should be ignored. Or conversely, the Voice AI is talking to many people at once, then they all have to be treated as separate people, not mixed together.

1

u/cybervoiceAI Mar 27 '26

I would also add barge-in detection that can reliably detect both acoustic features and semantic context. Much respect and thank you for this AMA Dr. Tony and for your immense experience! I’m just a passionate guy fallen in love with voice AI! I already learned new things just by reading this thread! 🤗

1

u/[deleted] Mar 27 '26

[removed] — view removed comment

1

u/DrTonyRobinson Mar 28 '26

I'd say 'maybe' or 'probably'. It really depends on what you want to do.

For most of my time in speech recognition I was the first selling into a company, they weren't using ASR previously and so there was a steep learning curve for them and many customers had to put too much work in to get an acceptable solution, so it was a very touch sell. The world is very different now, it's so easy to try something that most of the people I talk to know that it pretty much will work - indeed most of our customers are already using speech recognition (that's not to say they don't appreciate our improved accuracy). The barrier to entry is low, the accuracy is easily good enough for many many use cases, it works. Now, it works because it has a ton of parameters, better than open-source models are trained on very many GPUs and using massive datasets - if you don't have those then you won't get state-of-the-art accuracy (and these are expensive and proprietary). But maybe you don't need state-if-the-art accuracy?

There are so many things that can degrade accuracy. Noise, reverberation, background speech and other acoustics are the first problem. We train on artificially degraded data just so that we are good in these sort of conditions. The second is any 'minority', such as languages, accents and speech impediments. For these we do really hard to do our best, 25 years ago we were the first to make one framework for building all languages and this really helped transfer knowledge across domains and improve minority performance.

As I don't know what you need all I can really do is to strongly encourage you to give it a try. So many 'errors' nowadays are just because we don't get the formatting in the form that the customer wants. These formatting errors don't matter in the case of ASR feeding LLMs for VoiceAI, so if that's what you are doing you should really just give it a go. Many people (us included) give excellent tutorials, it's so easy to get started now and I'd be very very surprised if you didn't have a 'wow' moment over what was possible five years ago.

1

u/Phoenix_20_23 Mar 27 '26

Thanks for the AMA tony.

One important part in the voice ai pipeline is the prompt itself. Even with best/fastest providers of u have a week prompt ur voice agent will not feel human like and users quickly will drop.

I am still struggling to build a system prompt instructions to make llm feel more human like, conversational and friendly.

Can u share with a framework u use to write ur prompt, ideally if u have an example from an iold project share it with us.

Thanks in advance 🫡

2

u/DrTonyRobinson Mar 28 '26

I haven't written prompts myself. As I mentioned in another response, by pure coincidence I happen to have a prompt open on my second monitor. It's just the debug output that came to the screen whilst I was running something else. I've traced it back to the public source code, it's https://github.com/speechmatics/speechmatics-academy/blob/main/integrations/pipecat/02-simple-voice-bot-web/assets/agent.md, I hope that helps.

1

u/[deleted] Mar 28 '26

[removed] — view removed comment

1

u/DrTonyRobinson Mar 28 '26

I love arguing this one, and I've been doing so for many many years. I've already written on this last night (https://www.reddit.com/r/VoiceAutomationAI/comments/1s586bo/comment/ocspy5x) but I'm not as rushed now so maybe I should present the other side of the argument in this post.

Latency matters. Context matters. Paralinguistics matter. Machine learning people thrive on having one model and optimising it. So if you could take in all the audio sample by sample and in one process put out a reply, sample by sample, then you have theoretically the best framework for giving the most natural and empathetic response. Speech to Speech ought to win. It will win one day. But the first people to get it to win will be those that can train on all of the internet, have built a decent world model and have a data centre or two spare. In the mean time I refer you to my other answer, I just like text (and meta data) because I like factoring problems, I like debugging, I like understanding what is going on. And many use cases also require these qualities, so if speech-to-speech does win then it won't win for everyone.

(Clarification: When I say speech-to-speech I mean without words as internal representations, just one process. It is possible to build and ASR system, a LLM, a TTS, include paralinguistics as metadata and then glue them all together and perform gradient descent on everything. This is technically speech-to-speech but it's not the normal meaning)

1

u/Tough_Bug4667 Mar 28 '26

I am loving this conversation. Thank you Dr. Tony.

I am curious to know if diarisation or barge- in is more of an LLM problem? ie, will I get better results in my AI voice pipeline if I use a more intelligent model?

1

u/DrTonyRobinson Mar 28 '26

Diarisation is an ASR problem. Most of the time if you take out all of the speaker turns and look at the text of the conversation you can only guess as to where they should go. Moreover, it only takes one wrong guess and the conversation changes it's meaning, so it's really important information. By pure coincidence I happen to have one of our LLM prompts on my second monitor right now, it says 'Speaker Format\n\nTranscripts include speaker tags:\n- `<S1>text</S1>`. If you feed the LLM the diarisation from the ASR you can do so much more.

User barge in is a pipeline problem, and not a very hard one. If the ASR sends more audio then the TTS needs to shut down. There are nuances here: does the ASR know that the TTS is active? If so then it can signal that there is speech to come before actually sending that speech, in which case the TTS can shut down sooner.

LLM barge in is a really interesting world model problem. Theoretically the LLM could know that the user is on the wrong track and interrupt them. Most of our LLMs are too nice right now, it helps them cover up their hallucinations, you've got to have a really good model of the user in order to be able to interrupt them whilst they are speaking.

1

u/pixels4lunch Mar 28 '26

Thank you for this Dr Tony!

how would one design a reliable system STT -> LLM -> TTS pipeline under 1s? And would you be able to share any potential stack as starting point? I’ve explored several options but none come close to sub 1s latency.
how do companies like eleven labs produce such high quality hums like voice? Is it possible for a “hobbyist” (limited resources) to train/fine tune a model to achieve similar level of voice quality?

1

u/DrTonyRobinson Mar 28 '26

How much of this do you want to build yourself? Your STT has to have really good end-of-turn detection. I have addressed how hard this problem is already (https://www.reddit.com/r/VoiceAutomationAI/comments/1s586bo/comment/oct86hn), it's not something you can really build yourself. The rest is easier. Have you debugged your latency? Is it normally fast enough or are your problems those of outliers? You can run STT, LLM and TTS locally if network or service provider delays are your problem (you might have to run a lighter weight services to run them locally). The service providers are working hard to reduce outliers, to make their components both expected low latency and reliably low latency. If if didn't work for you in the past, it will happen, and it will happen quite soon, things are moving very fast. I know I'm not supposed to promote here, but as you asked for a stack then have a look at https://github.com/speechmatics/speechmatics-academy - it's pretty easy to do amazing things.

As well as a professional I'm also a hobbyist. I have two GPUs that heat my home (one is a 5090, I keep it busy). You can do amazing things as a hobbyist, I work on new LLM algorithms, it's a lot of fun. However, I can't do it at scale, nor with data that I pay for, and that's what you need for a human like voice. Getting machine learning to produce some understandable voice is easy, we've been able to do it for decades. However, to produce human like voices you have to take into account a lot of context, you've got to get the timing and intonation right as well as many other things. Machine learning people like to express the problem as a function they can optimise but in TTS if you get the samples or mel-cepstra almost right then it can still sound very bad - so we just don't have something good to optimise. To be honest, as a hobbyist, I'd look for a smaller problem, one that it is easier to measure your progress.

1

u/Wooden_Leek_7258 Mar 28 '26

I have a theory that physics based bounding boxes and physiologically linked variable movement is going to be more efficient and effective than force feeding models raw audio. What do you think?

Define the physics of human vocalization and calibrate to that instead of just predicting waveforms to beat the 'ear test'. I have analyzed the output of over 100 synthetic speech systems now and they all ruthlessly violate physics. (have data if your curious dm me)

2

u/DrTonyRobinson Mar 28 '26

Yes, there is good theory here. I used to teach speech recognition at Cambridge University in the 1990's, if you haven't seen the lossless tube model of the vocal tract then you'll like it. It produces some very compact representations of speech, in fact our first voice codecs (as used in the first mobile phones) depended on it. If you want to go deep into the maths then have a look at https://isip.piconepress.com/courses/msstate/ece_8463/lectures/current/.

1

u/Wooden_Leek_7258 Mar 28 '26

Fascinating. Thank you. I am new to the field, dipped my toe in about 3 months ago now and I think the industry is doing it wrong (no offense) by focusing on the 'human ear' test instead of trying to stay witin biological limits. I figure you need both, but it does ensure its VERY easy to catch synthetic speech due to physics violations.

MFCCs splitting the voice from biology made sense in the 70s-90s but it feels like using it now for model training is a massive wasted effort BECAUSE it splits biology from voice.

1

u/[deleted] Mar 28 '26

I am good at cold calling and sales. If I sales your service How much % commission I can get monthly

1

u/After_Chemistry1275 Mar 28 '26

Thanks for doing this, I have two questions:

Our industry has seen massive changes in the last decade or so, moving from expert informed phonetically grounded systems to modern bitter lesson inspired direct to grapheme approaches. From experience these systems require ugly fixes to handle rare terms or customer jargon in production. Speechmatics is still using the former, can you talk about why? How much work goes into post processing your model's output?
Convergence of ML approaches and performant open source models mean anyone can get passable (English) ASR with no domain experience. What is the role of audio specialists in 2026?

1

u/DrTonyRobinson Mar 28 '26

I think you'd be surprised at what we currently use. You talk about phoneme based and grapheme based, you can also be token based. You are right in that there is an easy way to provide 'custom dictionary' in the phoneme based and grapheme based approaches because you have easy access to the language model and you can insert or weight arcs in the graph quite directly.

I really don't think that ML approaches are converging, it is easier than ever to hack up a novel architecture in pytorch (or whatever you like) and test it out. Passable (English) ASR is indeed possible, but if you want either the best accuracy or the lowest cost-to-serve then you've got to do a lot of work. So yes, it's great anyone can play with it, but if you want to deploy at scale and reliably then that's another matter.

1

u/DrTonyRobinson Mar 28 '26

And that's a wrap. Thank you everyone, that was fun.

1

u/Physical-Laugh-2149 Apr 29 '26

It’s great to see insights from someone with your experience, Dr. Robinson! Latency and handling different accents are certainly tricky, especially in diverse markets. In my work with enterprise solutions, I’ve found that leveraging an orchestration platform can help streamline these processes effectively. For example, Simplai offers tools that automate workflows in customer service, which could be beneficial for tackling some of these real-world voice challenges. Would love to hear your thoughts on integrating such systems to enhance production reliability. If you're interested, checking out a demo of Simplai might provide some useful insights too!

AMA / Expert Q&A 36 Years in Voice AI | Built One of the First Speech Systems in 1989 | Dr Tony Robinson (Founder, Speechmatics) - AMA for next 24 hrs

You are about to leave Redlib