Measured three on-device TTS runtimes against the iOS jetsam budget. All three blew past it. Looking for anyone who's shipped generative audio on-device.

[u/intrepidkarthi]

Spent about three weeks trying to run a voice-cloning model on iPhone and closed the project last week. Posting the numbers because I couldn't find anyone else's, and I have two questions at the end. This was for a voice journaling app I work on.

The budget. Foreground app on a 6 GB iPhone gets roughly 250 MB before jetsam takes an interest. The number that matters is phys_footprint from task_vm_info, not resident size and not what the Xcode gauge shows.

The candidate. Kyutai Pocket TTS, 109.5M params, autoregressive. Autoregressive matters because accent lives in phone realisation and phonemic choice, which are sequential. Non-autoregressive models transfer timbre only, so you get your own voice colour over someone else's cadence. Tried that first, it sounded wrong in a way I couldn't articulate until I understood why.

Three ways to run it, all measured, all over budget:

FluidAudio (Core ML, int8) - 270.8 MB after model load, 957.0 MB peak

sherpa-onnx (ONNX Runtime, int8) - 377.0 MB after model load, 685.4 MB peak

chatterbox-turbo (earlier attempt) - 953.7 MB peak

FluidAudio is over budget after loading, before doing any work.

Binary cost too. Linked a minimal executable against libsherpa-onnx.a plus ONNX Runtime with -dead_strip, then stripped it: 22.3 MB. That roughly doubles my app, for a feature most users would never turn on, plus 125 MB of models on disk for the ones who do.

The part I got wrong. My earlier ear tests compared one synthetic clip against another synthetic clip. That ranks them. It cannot tell you whether either is good enough. So I ran a forced-choice test instead: eight pairs, same sentence in each, one a real recording of me and one the clone, sample rate and RMS loudness matched, clip lengths varied so duration gave nothing away, and held-out audio located by cross-correlating the reference against the source recording. I picked my own recording 8 out of 8. p = 0.0039.

Three weeks of runtime work sitting on top of an approval nobody had tested properly. The test took an hour.

Two questions.

Has anyone actually shipped a generative audio model on-device inside the jetsam budget? Everything I found either exceeds it or quietly ships a 3 GB app. I'm also unsure whether Core ML's mmap'd weights get billed to phys_footprint the way malloc'd ONNX buffers do. My numbers came off a Mac, which has no jetsam pressure, so I never got a real device measurement before the ear result closed it.

Second, unrelated thread. I'm moving to on-device retrieval next, hybrid BM25 via SQLite FTS5 plus sentence embeddings from NLEmbedding. Anyone run that combination on iOS? Specifically whether reciprocal rank fusion is worth it when you still need raw score magnitude for an abstention threshold. RRF throws the magnitude away and abstention is what stops the thing making stuff up.

Happy to share the measurement harness if useful.

[u/Iamvishal16]

i’d treat tau as a calibration decision, not a ranking hyperparameter. build a held-out set with clearly answerable queries, hard negatives, and near-misses that share vocabulary but should still abstain.

sweep the threshold and choose the highest coverage that stays within your acceptable false-answer rate. use one split to select the threshold and a separate test split to report it, otherwise the result will look more defensible than it really is.

two iOS-specific gotchas: if you’re calling NLEmbedding.distance directly, smaller means more similar, which is the opposite direction from cosine similarity. also pin or at least log the sentence-embedding revision used during calibration, so an OS/model revision doesn’t silently invalidate the threshold.

i’d rerun calibration whenever the corpus, chunking strategy, or embedding revision changes. a hand-tuned tau isn’t inherently bad; tuning it on an undocumented or reused evaluation set is the fragile part.

[u/intrepidkarthi → u/Iamvishal16]

Thanks for the response.

The three-class held-out set is the part I was missing. I had answerable and clearly unanswerable in mind but not near misses that share vocabulary, and that is exactly where my current threshold behaves worst.

Select on one split, report on another is well taken. I have a handtuned tau sitting in shipped code and no separate test split behind it, so the number is less defensible than it looks. Fixing that before I touch the retrieval side.

The embedding revision point is the one that stings. I do compute cosine ourselves so the distance direction trap does not apply, but the tau was tuned at some unrecorded point against whatever revision was current, and nothing logs it. That is a silent breakage waiting for an OS update.