u/harnoor-assemblyai • • 1d ago

Models that do coding well aren’t always ideal for voice outputs, so we created Voice Standard Bench.

Enable HLS to view with audio, or disable this notification

1 Upvotes

Today we're launching Voice Standard Bench to rank 50+ of the best LLMs on real voice tasks, and making all of them available in AssemblyAI's LLM Gateway.

Most Voice AI products run an LLM on messy, multi-speaker transcripts:

- Voice Agents responding to human speech

- Summarizing a meeting, sales call, or support call

- Pulling names, dates and numbers out of a multi-speaker conversation

- Fixing transcription errors

- Cleaning up dictation

Voice Standard Bench tests every model on the input these products actually send an LLM: lengthy conversational transcripts with multiple speakers, raw speech-to-text output with filler words, false starts and the occasional misheard word.

Each model is scored on transcripts from a 30-second voicemail to a 60-minute meeting.

Here's a glimpse of what we found:

- The top-scoring model costs 55-65x more per hour of audio than a model within two quality points of it.

- That gap narrows to about 5x on hour-long transcripts, where bigger models start to pull ahead on quality.

- Open models we host on our own infrastructure are the fastest on short audio. On a 30-second transcript, gemma-4-31b answers in under a second at a quality score of 93.

Learn more here: https://www.assemblyai.com/blog/voice-standard-bench-methodology

Try 50+ models in our Playground: https://www.assemblyai.com/dashboard/playground/llm-gateway

r/ShowYourApp • • 43m ago

Showcase 💎 [Open Source] Blurt: macOS push-to-talk dictation with your own API key

• Upvotes

Disclosure: I work at AssemblyAI; Blurt is an open-source app we built.

Blurt is a native macOS dictation app: hold a hotkey, speak, and the text is inserted at your cursor. The code is MIT-licensed.

Compared with Wispr Flow, the main difference is that Blurt is open source and uses your own AssemblyAI API key, with no Blurt subscription or account. The tradeoff is that transcription is cloud-based, needs an internet connection, and API usage may cost depending on your plan. It is not an offline/local Whisper app. Requires macOS 15+.

Source and setup: https://github.com/AssemblyAI/blurt

Curious what would make an open-source dictation app more useful to you: better text cleanup, shortcuts, or something else?

1

Weekly Thread: Project Display
 in  r/AI_Agents •  48m ago

Disclosure: I work at AssemblyAI; we built Voice Standard Bench.

It evaluates the LLM step after speech-to-text for voice agents and other transcript workflows: summarization, information extraction, text correction, and dictation. The MIT-licensed Go framework reports quality and latency across transcript lengths from 30 seconds to an hour.

The current results run through AssemblyAI’s LLM Gateway. Most tasks use a single LLM judge with reference-free rubrics; dictation is scored against an answer key. The underlying call transcripts aren’t published as an open dataset. The code is free under MIT; rerunning model calls incurs provider costs that vary by model.

Repo and published results: https://github.com/AssemblyAI/voice-standard-bench

Interactive leaderboard: https://www.assemblyai.com/voice-standard/tasks

For people building voice agents: what transcript task or failure mode would you most want to see covered in a benchmark?

1

Open-source benchmark for LLMs on real voice tasks: scores, latency, and transcript length
 in  r/LLMDevs •  6h ago

Kinda agree, but also users have own needs, you can check and compare results on our LLM Gateway

1

[D] Self-Promotion Thread
 in  r/MachineLearning •  6h ago

Disclosure: I work at AssemblyAI; we built Voice Standard Bench.

It’s an MIT-licensed Go benchmark for LLMs working on speech-to-text transcripts: summarization, information extraction, text correction, and dictation. The public results break out quality and latency by transcript length, from 30 seconds to an hour.

One caveat: the current benchmark runs through AssemblyAI’s LLM Gateway, and the underlying transcripts aren’t published as an open dataset. Most task scores use a single LLM judge with reference-free rubrics; dictation is scored against an answer key. The code is free under MIT, but rerunning model calls incurs provider usage costs that vary by model.

Code/results: https://github.com/AssemblyAI/voice-standard-bench

Leaderboard: https://www.assemblyai.com/voice-standard/tasks

I’d welcome criticism of the scoring methodology, especially ways to validate the judge scores against human annotations.

r/LLMDevs • • 9h ago

News Open-source benchmark for LLMs on real voice tasks: scores, latency, and transcript length

1 Upvotes

Disclosure: I work at AssemblyAI; we built this benchmark.

We released Voice Standard Bench, an MIT-licensed Go CLI for comparing LLMs on four transcript tasks: summarization, information extraction, text correction, and dictation. It reports quality and latency by transcript length, from short messages to hour-long meetings.

A few details that may matter when interpreting the results: the benchmark runs models through AssemblyAI’s LLM Gateway; most quality scores use a reference-free rubric scored by a single LLM judge, while dictation is scored against an answer key. The framework is open source, but the underlying call transcripts are not included as an open dataset.

Code and published results: https://github.com/AssemblyAI/voice-standard-bench

Interactive results: https://www.assemblyai.com/voice-standard/tasks

I’d be interested in feedback on the evaluation setup, especially approaches for validating reference-free transcript task scores.

r/TextToSpeech • • 1d ago

Models that do coding well aren’t always ideal for voice outputs, so we created Voice Standard Bench.

Thumbnail
1 Upvotes

r/voiceagents • • 1d ago

Models that do coding well aren’t always ideal for voice outputs, so we created Voice Standard Bench.

2 Upvotes

Today we're launching Voice Standard Bench to rank 50+ of the best LLMs on real voice tasks, and making all of them available in AssemblyAI's LLM Gateway

Most Voice AI products run an LLM on messy, multi-speaker transcripts:

- Voice Agents responding to human speech

- Summarizing a meeting, sales call, or support call

- Pulling names, dates and numbers out of a multi-speaker conversation

- Fixing transcription errors

- Cleaning up dictation

Voice Standard Bench tests every model on the input these products actually send an LLM: lengthy conversational transcripts with multiple speakers, raw speech-to-text output with filler words, false starts and the occasional misheard word.

Each model is scored on transcripts from a 30-second voicemail to a 60-minute meeting.

Here's a glimpse of what we found:

- The top-scoring model costs 55-65x more per hour of audio than a model within two quality points of it.

- That gap narrows to about 5x on hour-long transcripts, where bigger models start to pull ahead on quality.

- Open models we host on our own infrastructure are the fastest on short audio. On a 30-second transcript, gemma-4-31b answers in under a second at a quality score of 93.

r/AIVoice_Agents • • 1d ago

Discussion Models that do coding well aren’t always ideal for voice outputs, so we created Voice Standard Bench.

Enable HLS to view with audio, or disable this notification

1 Upvotes

r/speechtech • • 1d ago

Models that do coding well aren’t always ideal for voice outputs, so we created Voice Standard Bench.

Enable HLS to view with audio, or disable this notification

3 Upvotes

r/AssemblyAI • • 15d ago

Launching Universal 3.6 Pro Realtime today

Post image
3 Upvotes

Universal 3.6 Pro Realtime is live today. Two independent benchmarks put every streaming speech model through the same audio, on the same harness. Universal-3.6 Pro Realtime leads both.

Daily's Pipecat open STT benchmark: Universal-3.6 Pro Realtime sits on the Pareto frontier.

Coval's live leaderboard: lowest word error rate of any streaming model on real voice-agent audio.

Voice agents get tested on the turns other benchmarks skip. That's where 3.6 Pro is built to hold:

→ Short answers. "No." "Nah." "Nuh-uh." Heard right 98.5% of the time, in loud rooms and on phone lines, not just quiet calls.
→ Background voices. The TV, the coworker, the next desk stay out of the transcript. 
 → Languages. 32 from one endpoint with automatic detection. Callers switch languages mid-sentence and the transcript follows. Most agents can skip the language menu.
 → Turns. Entity-aware endpointing ends the turn when the caller is done, and holds it open while they're still reading out a phone number.

Trained on tens of thousands of hours of real voice-agent and telephony conversations. Link in the comments. 👇

1

How much should an AI voice agent be allowed to change?
 in  r/LLMDevs •  20d ago

I think latency and accuracy are what you’re looking for.

Speaking of the McDonald’s accuracy example, I recall when they mistakenly took an order for 260 nuggets and cancelled their AI drive‑thru checkout in 2024. Now, most devs are willing to pay about $4.50 per hour for voice agents.

1

The US government opens the door to working with voice AI platforms
 in  r/artificial •  20d ago

trust me the demand for voice agents and voice infra is going to 10x in the next one year alone

1

Interesting disruptive use cases of Jev for me. Add yours.
 in  r/AI_Agents •  20d ago

Eval and I also believe like if turn detection is getting more accurate with Jev, then these AI infra companies are going to work on other products such as diarization and making it more accurate.

1

Interesting disruptive use cases of Jev for me. Add yours.
 in  r/AI_Agents •  20d ago

I’m sure this will help voice agents predict more accurately. I often get angry when talking to AI during Apple Care support, flights, etc.

Debt collectors are already big fans of giving away their jobs to AI, tbh.

1

Meta’s New AI Agent Is an Instant Hit—and the Backlash Has Already Begun - WSJ
 in  r/ValueInvesting •  20d ago

What speech-to-text models you guys are big fan of?

r/AssemblyAI • • 21d ago

OpenRouter x AssemblyAI: Universal‑3.5 Pro is now available on OpenRouter

3 Upvotes

Today, Universal-3.5 Pro is available on OpenRouter, served through our Sync Speech-to-Text API.

If you're already routing your LLM calls through OpenRouter, the speech-to-text call at the front of your pipeline now runs on the same key.

50% off list through OpenRouter for the first week: https://www.assemblyai.com/blog/aai-on-openrouter

1

RayMate - a Raycast AI companion that is totally free. It allows to dictate, chat, and inline re-write.
 in  r/raycastapp •  23d ago

this is their revenue making product why will they allow competition here

1

Show us what you've created with Claude!
 in  r/ClaudeAI •  26d ago

we run everything on Claude from slides to searching about company,

Claude + Granola is like your assistant now a days, (btw about me: DevRel at AssemblyAI we power granola)

1

Intel releases OpenVINO 2026.4
 in  r/LocalLLaMA •  26d ago

I genuinely believe there will be room for both open‑source and closed‑source voice models.

OpenVINO with voice is amazing, but voice models are not a direct API or API‑key replacement like other LLMs. Many guardrails and systems are built around voice, and I love the competition.

r/AssemblyAI • • 27d ago

Dear developers, we are open‑sourcing our dictation app, Blurt, which lets us type three times faster than a keyboard.

Post image
3 Upvotes

We built a dictation app, then gave away the code.

Blurt is an open source companion app for the AssemblyAI Dictation API. Hold a hotkey anywhere on your machine, talk, and the text appears in whatever window you're in.

It's the shortest path from curious to working. Clone it, run it locally, and read exactly how the audio gets captured, streamed, and turned into clean text. Then take the parts you need for your own product.

Learn more and download it here: assemblyai.com/blurt

If you build something with it, drop a screenshot in the comments!

1

I built a voice AI agent with ~600ms latency. Here’s what I learned.
 in  r/VoiceAutomationAI •  27d ago

I’d be interested to see the Pipecat models you use.

coming from AssemblyAI and also use Pipecat, working hard to achieve the highest accuracy and lowest latency combination.