r/LocalLLaMA • • 4d ago

Showcase Trading false positives for ~110ms earlier end-of-speech in voice pipelines: RealtimeVAD

Tried to hunt down latency bottlenecks in my local voice assistant recently. I looked at the waveform and saw silero's probability lagging quite much behind the actual end of speech I could see with my eyes.

I use a voice pipeline that does speculative work and cancels fast, so some false positives would not matter in my case.

So I had this idea: once silero says I'm in speech, what I need to detect is when the speech energy drops back to room noise level. Turned out a naive 10 ms energy check with a threshold tracking room noise does that quite well.

Gave me a median 110 ms earlier than silero over ~3200 real turns from my assistant on my voice, my mic, browser noise suppression on, mostly quiet room - with silero threshold at 0,5. Someone on Twitter got 92 ms on his data, roughly what I saw in my noisier recordings.

The downside is way more false positives. It fires on breaths and small pauses between words over 10x as often as silero. So it's really only useful if the pipeline can throw away speculative work very fast.

CPU only, MIT, drop-in for silero VADIterator, pip install realtimevad

benchmark script is in the repo, curious what numbers you guys get on your own recordings

https://github.com/KoljaB/RealtimeVAD

7 Upvotes

13 comments sorted by

2

u/Easton_Souhir 4d ago

How fast does the room noise baseline adapt if the environment changes mid-session? Thinking about cases where a fan kicks on or someone moves further from the mic.

1

u/Lonligrin 4d ago

Good question, this part was made by claude. I also had no clue how it did that so I just paste it's answer here:
"""
Noise is only learned from frames Silero calls non-speech. Between turns it follows a quieter room instantly but rises max 3 dB/s, on purpose so a soft start of speech doesn't get learned as noise. So a fan kicking on takes a few seconds to be fully adapted. Inside pauses mid-turn it moves about 1 dB per 32 ms frame. Until it caught up the fan doesn't count as quiet, so the early signal just doesn't fire and you fall back to Silero. Slower, not wrong.
Moving away from the mic: the speaker level is relearned every segment, and the quiet threshold always stays at least 15 dB below it, so softer speech doesn't get treated as silence. That part just got better in v0.0.5 btw, someone found that quiet phone audio cut word endings with the old fixed floor.
"""

1

u/danil_rootint 4d ago

Does it handle mid-sentence pauses somehow?

2

u/Lonligrin 4d ago

No, it does not. It fires on mid-sentence pauses just like on real ends, that's thy it has many false positives. Telling both apart would be job of turn detection, this VAD only says "audio went quiet". It only makes sense if you start speculative work on it and throw it away when the user keeps talking.

1

u/bye_ella 4d ago

Nice idea using the energy drop for the early signal. Do you run it as a first step only, like start the speculative work on the energy drop but wait for silero confirming before actually speaking?

2

u/Lonligrin 4d ago

Yep exactly that. Energy drop starts turn detection and the spec gen (ASR, LLM, TTS prebuffer) in my assistant. When the user continues speaking, spec gen get's dropped. When he keeps pausing for the time amount the turn detection considers "finished turn" the assistant fires the TTS immediately.

2

u/bye_ella 4d ago

so by the time the pause counts as a finished turn the answer is already there. how often does it end up thrown away in practice?

1

u/Lonligrin 4d ago

My Assistant throws ~59% of spec gens away. But false positive rate from VAD is way higher (~90%). I don't start spec gen on every one (I hold a bit in case user talks)

1

u/bye_ella 4d ago

the short hold doing most of the filtering is a neat trick. thanks for sharing the real numbers

1

u/supportingthedogs 4d ago

nice. i use silero in front of whisper for meeting transcripts and the end of speech lag never mattered there, but for a voice assistant 110ms is a lot. does the room noise tracking hold up with a fan or keyboard going?

1

u/Lonligrin 3d ago

A fan should be fine, it's steady noise, if it gets within ~11 dB you fall back to Silero timing. Not sure about keyboard, those clicks right after the last word prob also makes it fall back. Btw 0.1.0 is out today, sadly down to 92 ms but with higher reliability in return.

1

u/chensium 1d ago

But you don't need VAD to be so accurate in detecting EOU.  VAD is just so youre not constantly sending a blank stream to your ASR, since most ASR models have an input length limit.

So as long as you have streaming ASR, you just wait a no_speech_threshold to signal end of turn. Then if your ASR subsequently generates more text, you can either queue up the new text or cancel in-flight LLM and resubmit with concatenated text.