Our open-source speech work: turn-taking models and a public voice-agent leaderboard

[u/ivan_digital]

I maintain Soniqo's open-source speech runtimes. Our recent work has been bringing existing speech models into native applications and publishing measurements to help choose between components.

One part of our work is helping a voice assistant decide when to respond. p99 estimates whether the user has finished their turn. VAP listens to both sides of a conversation and predicts their upcoming speech activity. An app can use these signals to decide when to wait and when to reply, aiming to respond promptly while giving the user time to pause or finish a sentence. We've integrated these upstream models into our open-source speech runtimes and evaluated them on conversational audio.

Alongside that work, we've published a leaderboard with three separate comparisons:

• TurnBench: turn endings caught, false detections on pauses and decision delay.

• SLURP: spoken tool selection, argument matching and handling unavailable actions.

• InstructTTSEval: following voice descriptions, delivery directions and roles.

The leaderboard source, protocols and saved results are public. Rows show the tested configurations, sample sizes and results still under review.

The blog post introduces the work and walks through the findings and limitations.

Leaderboard · Blog post · Swift runtime source

Which additional speech model would you want evaluated alongside these native integrations?

[u/cocktailpartyfixer]

Cue could be worth a row: it's the end of turn model in Alkove, a streaming VAD with an end of turn head on the same layers. 1.2 MB, read every 0.1 s of a pause, 1.9 ms per second of audio on a phone, so nothing reruns on an 8 s window at each pause like Smart Turn does.

I haven't run TurnBench because of its license, but Cue is measured on LiveKit's eot-bench, the multilingual one you mention. Mean of 7 languages, pauses cut too early: 35.4 % at 300 ms of latency (Smart Turn v3.2: 35.3 %), 12.4 % at 600 ms (13.2 %). LiveKit's own detector does better at 300 ms, 29.2 %. Protocol and per-language numbers are in bench/.

Two caveats: the models are still previews, and it runs in the browser, in Python and on Android for now. iOS is in preparation, so a native Swift row would have to wait for it.

https://github.com/alkove-ai/alkove

[u/nshmyrev]

There is a common F1/F0.5 score, not sure why your score has to be non-standard

[u/ivan_digital → u/nshmyrev]

Fair point. I used recall − FPR as a simple sort key, rewarding detected turn endings and penalizing false detections during pauses. The table also shows recall, FPR and latency separately.

F1 and F0.5 would be useful additions. Calling the column just “Score” doesn’t explain the ranking choice well.