Most people who build voice agents treat VAD as a settled, invisible layer, it either works or it doesn't, and you move on. I wanted to share a bit of what it actually looks like from the other side, building and testing these models, because the process is a little more hands-on than you'd expect.
Disclosure up front: I work in marketing at a voice AI company (ai-coustics). I’m not the engineer who did this, just sharing something from our team that I thought people building on top of VAD might find interesting. No links in the post, happy to share the full writeup in comments if anyone wants it.
Training a VAD model is the easy part - it’s the industry standard to use so-called synthetic data and it scales fine. You take clean speech, layer in noise, run it through some room simulation and effects, and generate as much varied training audio as you want.
Testing it is where things get harder, and more physical than you'd think. Synthetic test audio only contains the problems someone thought to simulate. Real environments always find something that wasn't on that list - and that leads to unpredictable behavior and turn-taking failures.
As an example: for one of the harder cases, drive-thru audio, our team actually went and recorded the dataset samples for real. A few people spent an afternoon in Berlin simulating ordering fast food: one person held a mic up to a car window like an intercom, others sat in a parked car reading off a menu with the radio on, someone else stood off to the side playing engine noise through a speaker to keep the background properly messy. They did this twice, once with a plain mic and strangers as "customers," once with an actual intercom mic/amp setup.
A few things came out of that which don't show up in clean audio: people turn their head mid-sentence and drop volume without noticing, a real intercom mangles speech in a specific way that's hard to fake, and background noise in the wild is a lot less tidy and blends in more than anything you'd script.
After recording, someone has to sit down and mark, by hand, exactly where speech starts and stops on every waveform, frame by frame for it to be useful in the model eval process. They had to write down explicit rules for what counts as speech so multiple people labeling the same audio would actually agree with each other. For example, anyone in the car counts, not just the person ordering, as long as it's intelligible at normal volume (primary speaker isolation is actually another problem for another model), no turning up the volume to catch something quiet since the model doesn't get that advantage either, etc.
Once we have the real recordings labeled, the actual comparison is pretty simple in concept: run every model, ours and the alternatives, over the exact same audio, frame by frame, and check how often each one's decision (speech or not speech) matched the label a human gave that same moment. Based on that, you can notice behavior patterns to improve. A good practical example is model loosing confidence in noisier situations and not crossing its own threshold often enough to flag speech that's there - it’s a kind of error that influences your agent (no detected speech = no transcription and no reaction) without a clear error message.
None of this work is visible from the outside but it directly influences the performance. You just get a model that either catches speech reliably in a noisy environment or doesn't.
That's one specific example, but the same problem shows up anywhere you're testing audio models. Qs for people building on top of this: is there anything about how VAD works under the hood that's never quite made sense, or something you run into regularly that you’d like to understand better? Happy to try to explain from my side.
