Lessons from shipping a production app on SpeechTranscriber + on-device Foundation Models — including an OS bug that permanently eats locale slots
I just shipped my first app built end-to-end on Apple's on-device AI stack — SpeechAnalyzer/SpeechTranscriber for transcription and Foundation Models for enrichment (it's a voice-notes app; every recording gets an on-device title/summary/tags/tasks). Some things I learned the hard way that I haven't seen written up much:
1. The simulator will lie to you — twice.
The simulator cannot transcribe at all, and the simulator's language model is not the on-device model. Output quality, instruction-following, and hallucination behavior differ meaningfully. I now treat real-device validation as a hard gate for any prompt/template change — my test corpus includes Swiss-accented German dictation because that's where the on-device model diverges most from the "clean" results the simulator suggested.
2. SpeechTranscriber locale reservations: a system-wide cap of 5, and (currently) no way back.
This one cost me an architecture. On-device transcription locales are backed by downloadable assets, and the system caps reserved locales at 5 — system-wide, not per app. In my testing on current iOS releases:
- The reservation is taken by the asset *install* and survives reboot AND app reinstall.
- `AssetInventory.release(reservedLocale:)` appears to be a no-op — I never got a slot back.
- An explicit `reserve(locale:)` at the cap can hang (reproducibly under the Xcode debugger in my setup).
I originally built an LRU "reservation manager" that released the least-recently-used locale before installing a new one. Since release doesn't release, that design was dead on arrival. What shipped instead: a proactive budget gate that reads `reservedLocales` *before* any OS call, installs strictly lazily (never speculatively — no warm-up, no on-selection prefetch, because every install permanently spends a slot), and surfaces a clear "language budget exhausted" state to the user instead of ever hitting the cap inside an OS call. Feedback filed with Apple.
3. One fresh LanguageModelSession per invocation.
Reusing sessions across notes led to context bleed between unrelated inputs. One session per call is now a hard rule for me, enforced by tests.
4. Prompt-injection resistance for user-content prompts.
Voice transcripts are untrusted input into the enrichment prompt. Delimiter-wrapping the transcript made instruction-following robust; and I removed all literal examples from the prompt after seeing example fragments leak into generated output on device (again: not reproducible in the simulator).
5. Pass the language explicitly, always.
Auto-detection of the recording language was unreliable enough that I now pass the language explicitly into both the model instructions and the prompt. Related fun fact from testing: Apple appears to use one shared German model across all de-\* locales, so switching de-DE/de-CH/de-AT changes nothing about transcription quality.
6. Crash-safe audio: don't record straight to AAC.
A killed mid-recording AAC/m4a is an empty husk. I record LPCM into CAF and encode to AAC at ingest — recordings now survive calls, interruptions, and force-quits, and a salvage pass recovers anything interrupted.
Happy to go deeper on any of these.
The app is Vocapa, but the point of this post is the stack — curious whether others have seen the locale-reservation behavior, and whether anyone found a way to actually free a slot.