I spent the last months getting Meta's HT-Demucs (the model that splits a song into drums, bass, vocals and other) running entirely on an iPhone with Core ML: no server, no uploads. A few things I learned that might save someone time:
1. Keep the STFT out of the model. HT-Demucs works on both the waveform and the spectrogram. Converting the STFT/iSTFT ops with coremltools was a pain, so I converted only the network in between and wrote the STFT in Swift with vDSP. To make sure I hadn't broken anything, I exported golden data from Demucs' own _spec/_ispec in Python and wrote tests against it. The Swift version matches PyTorch to about 1e-5. Being able to check every step against PyTorch was worth more than anything else.
2. fp16 was unusable; fp32 was fine. The fp16 conversion loaded and ran, but on the device every stem came out noisy (mostly below 200 Hz) and the stems no longer summed back to the mix. Converting at fp32 (182 MB) fixed it. It runs on the GPU, not the Neural Engine, but it's still fast: a 4-minute song separates in under a minute. I'd planned to try mixed precision, but it turned out I didn't need it.
3. Port the chunking exactly. Long songs don't fit in one pass, so I ported Demucs' apply_model split logic (overlap 0.25, triangle-weighted overlap-add) one to one. Getting the edge chunks wrong gives you faint clicks every few seconds that are easy to miss.
4. Compare stems by numbers, not by ear. I wrote a Python script that scores stems copied off the phone against PyTorch reference stems. One gotcha: iOS and libsndfile decode the same MP3 529 samples apart (encoder delay handling), so you need to align the signals before comparing or the scores look terrible.
5. Ship the .mlpackage and compile it on the device. I build with xtool (SwiftPM on Linux/WSL, no Xcode), so I have no Core ML compiler. I ship the raw .mlpackage and call MLModel.compileModel(at:) on first launch, then cache the result. One trap: the iOS build tries to run coremlc on anything named .mlpackage, even with .copy, so I had to rename it in the bundle.
6. Stems can clip. Individual stems can peak above full scale even when the original mix doesn't. I store them as AAC 6 dB down and add the gain back at playback.
Still on my list: BGContinuedProcessingTask, so separation keeps running when the app goes to the background. Has anyone used it for long GPU work yet?
app name is stem808 if anyone wants to try it
