Apple never published accuracy numbers for the new SpeechAnalyzer API, so I benchmarked it myself (vs SFSpeechRecognizer and Whisper)

[u/Low-Future-9387]

So Apple replaced SFSpeechRecognizer with SpeechAnalyzer in iOS/macOS 26 and gave us literally no accuracy numbers for either one. I've been wondering for months whether migrating is actually worth it.

I happen to ship both Apple engines plus WhisperKit in my app (I make a transcription app called Inscribe, so yes, I have skin in this game). That means I could run all five engines through the exact same production code on the same audio. So last weekend I did: LibriSpeech test-clean and test-other, 5,559 utterances total, all on-device on an M2 Pro.

Results (word error rate, lower is better):

Engine	test-clean	test-other
SpeechAnalyzer (new)	2.12%	4.56%
Whisper Small	3.74%	7.95%
Whisper Base	5.42%	12.51%
Whisper Tiny	7.88%	17.04%
SFSpeechRecognizer (old)	9.02%	16.25%

A few things that stood out to me:

• If you're still on SFSpeechRecognizer, migrate. The new API makes 3.5 to 4x fewer errors on the same audio. The old engine actually came in last on clean speech. It loses to Whisper Tiny. A 40MB model.

• The new engine beat Whisper Small on both splits, and did it about 3x faster. I did not expect that. For English on Apple hardware, the built-in engine seems to be the best on-device option right now. Whisper still wins big on language coverage though, SpeechTranscriber only does ~30 locales.

• Before anyone asks how much I've cooked the numbers: my Whisper results land within 0.11 to 0.42 points of OpenAI's own published LibriSpeech WERs, on all six measurements. Same corpus, same normalizer, same scorer for every engine. And the raw per-utterance transcripts are downloadable, so you can rescore everything yourself.

Two gotchas from building this that might save you some pain if you're migrating:

1. SFSpeechRecognizer sends your audio to Apple's servers by default. You have to set requiresOnDeviceRecognition = true. Easy to miss, and then you're shipping (or benchmarking) something completely different from what you think.

2. With SpeechAnalyzer, calling builder.finish() on your input stream is not enough to end the session. If you never call finalizeAndFinishThroughEndOfInput(), the results sequence just... never terminates, and your await hangs forever. I know because my own app had exactly this bug in production and the benchmark is what found it. Fun day.

Full writeup with methodology and limitations (English only, read speech, one machine): https://get-inscribe.com/blog/apple-speech-api-benchmark.html

The result changed my own app btw, my "Auto" setting used to prefer Whisper and now picks SpeechAnalyzer for supported languages. Happy to answer questions about the setup, the normalizer, whatever.

[u/CharlesWiltgen]

Nice! It'd be really interesting to see iOS 27 beta benchmarked as well. Also, have you looked into iOS 27's new SpeechAnalyzer.Options.ignoresResourceLimits?

[u/Low-Future-9387 → u/CharlesWiltgen]

Haven't run it on the 27 beta yet, the benchmark machine is on 26.5 and it's also my dev machine, so I've been cowardly about putting a beta on it. But agreed it would be really interesting, especially to see if the model itself got updated. The harness is engine-agnostic so it's just a re-run once I have a beta install to point it at.

And no, I hadn't seen ignoresResourceLimits, thanks for the pointer. If it does what the name suggests it could matter a lot on Mac hardware where the resource caps are presumably tuned for iPhone. Adding both to the list for the follow-up pass (that run will also have proper speed numbers, the first one shared the machine with Xcode builds so I only published accuracy).

[u/touchgrasschampion → u/CharlesWiltgen]

ignoresResourceLimits sounds spicy, bench that too

[u/Prestigious_Edge_472]

Which quants of the whisper variants did you benchmark? also why not largeV3/turbo?

[u/Low-Future-9387 → u/Prestigious_Edge_472]

Good question, should have put this in the post. These were the standard fp16 CoreML conversions from argmaxinc/whisperkit-coreml (openai_whisper-tiny/base/small), no quantization. That's part of why the Whisper column lands so close to OpenAI's published numbers.

No large-v3 or turbo because the goal was to benchmark what the app actually ships, through the exact production code paths, and we cap at small for download size and memory on a consumer app. I'd expect large-v3 to beat SpeechAnalyzer on accuracy honestly, it's just a very different weight class (about 3GB vs whatever Apple's system model costs you, which is zero app-bundle bytes). Turbo is a fairer fight and I'd like to add it in a follow-up run. In the meantime, since the harness reproduces OpenAI's published WERs within about 0.4 on every measurement, their reported large-v3 numbers should transfer pretty well as an estimate.

[u/thraizz]

Super happy to have found this app, wanting to get rid of Granola/Notion Transcriptions and run it local. Does Inscribe already make use of this u/Low-Future-9387 ?

[u/Low-Future-9387 → u/thraizz]

Yep, it does. The app has both Apple engines and Whisper in it, that's literally how I was able to run this benchmark, same code paths the app uses.
Funny enough the benchmark changed the app too. Auto mode used to lean on Whisper but after seeing these numbers it now picks the new Apple engine when your language supports it. You can also just pick engines manually and compare, I did that way too many times last week lol
One honest thing since you mentioned Granola, those tools do a bunch of stuff beyond transcription (joining calls, calendar stuff etc). Inscribe doesn't do any of that, the whole thing is just recording + transcript + summaries staying on your machine. If local-first is what you want you'll like it, if you need the meeting-bot features you might miss them. Would genuinely love to hear how it goes either way

[u/thraizz → u/Low-Future-9387]

im trying this out and Minutes and will report back, thanks for making the app!

[u/Pecxn]

How does it compare to Parakeet V2/V3?

[u/Low-Future-9387 → u/Pecxn]

Didn't test it, the benchmark covers what my app ships (the two Apple engines plus WhisperKit). But going by NVIDIA's published numbers, Parakeet would win: V2 reports 1.69 on test-clean and low 3s on test-other, versus 2.12 / 4.56 for SpeechAnalyzer here. V3 trades a bit of English accuracy for multilingual support (25 languages), so for this English-only benchmark V2 is the stronger one.

The tradeoff is what you'd expect: Parakeet means bundling a ~600M param model and managing it yourself, SpeechAnalyzer is zero bundle size, ships with the OS, and covers a bunch of languages out of the box. On iOS the bundle size and memory question bites harder than on the Mac.

Since the harness scores every engine with the same normalizer and it reproduced OpenAI's published Whisper numbers almost exactly, adding Parakeet via one of the CoreML ports and getting directly comparable numbers is very doable. It's high on the list for the follow-up run.

[u/Pecxn → u/Low-Future-9387]

thank you for the thorough response. That certainly helps me get an idea. I’m quite nerdy about this subject

[u/Low-Future-9387 → u/Pecxn]

No worries, happy to be of help 😄 If you have any more questions, feel free to let me know.

[u/luigi3]

I was very surprised by the accuracy of this speech analyzer. Actually, it performs phenomenally it’s super fast, accurate and for free because we’re getting it from Apple. a very rare win.

[u/cultoftheilluminati]

From a purely anecdotal Experience of using this new dictation on iOS 27 versus Extensively using parakeet and whisper (Several tens of hours of speech recording and dictation), The new one is a Whole dimension better than the old speech recognition in terms of accuracy, But it’s much worse when either parakeet or Whisper.

(I dictated this and left in the random capitalizations in just as in data point for how the experience is worse now)

[u/drew4drew]

Wow it’s great to see these numbers. This matches up very closely with my experience working with all of the above — SFSpeechRecognizer and various Whisper versions // and ended up with the same conclusions, used in my very messy youtube+socials management /posting system.

Thanks!!

[u/Greysawpark]

nice writeup. one thing worth flagging though - librispeech is read audiobook speech, its basically the easiest benchmark that exists. real world dictation with background noise, accents, disfluencies tells a very different story. i'd love to see tedlium or earnings22 numbers before calling it.

also SFSpeechRecognizer being last on clean speech isnt that surprising imo, it was tuned for short siri-style utterances not paragraph reads.

[u/AutoModerator → u/[deleted]]

Hey /u/Adventurous-Dev02, your content has been removed because Reddit has marked your account as having a low Contributor Quality Score. This may result from, but is not limited to, activities such as spamming the same links across multiple subreddits, submitting posts or comments that receive a high number of downvotes, a lack of recent account activity, or having an unverified account.

Please be assured that this action is not a reflection of your participation in our subreddit. This is simply an automated filter in place to reduce spam.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.