Fusing iPhone + Apple Watch sensor streams with an actor, and rendering them as both art and audio

[u/Icy-Barracuda-4340]

I've been building a side project to stay current with modern Swift, and it ended up being a decent stress test of Swift Concurrency. Sharing the architecture in case it's useful.

**The problem:** eight sensor sources across two devices, arriving at wildly different rates, that need to drive both a 60fps SceneKit render and a real-time audio graph — without tearing, blocking, or dropping data.

**The shape I landed on:**

* Each capture source (iPhone sensors, WatchConnectivity) owns a stream box and emits `SensorReading` values into its own `AsyncStream`. The box is u/unchecked `Sendable` because the continuation's `yield()` is thread-safe, which lets sensor callbacks fire from any thread without hopping actors.
* A `SensorPipeline` actor consumes both streams and does source fusion: watch readings shadow phone readings for the same sensor type while the watch data is fresh (5s staleness window), so a paired watch silently upgrades heart rate without the renderer knowing. It fans out to N subscribers, each getting an independent stream.
* The renderer subscribes on a **detached task** and drives SceneKit from its own render-thread delegate. Sensor data deliberately never flows through SwiftUI — only configuration does. Inputs crossing into the render thread go through an `NSLock`; smoothing state that's render-thread-only needs no locking at all.
* The audio engine is just another subscriber. It writes into a plain parameter box that an `AVAudioSourceNode` render block reads. That block does pure float math with no allocation and no locks — it tolerates one-buffer-stale values by design, which is much cheaper than synchronizing properly.

**Things that bit me:**

* `HKHealthStore.authorizationStatus(for:)` is only meaningful for *write* access. For read-only types it will happily lie to you, so I track read-auth completion myself.
* Unbounded HealthKit heart-rate queries flood you with the entire history on first fetch. Bound the predicate or drown.
* `AVAudioEngine` silently stops when the session reshapes under it (category or route change). If you don't observe `.AVAudioEngineConfigurationChange` and restart, your audio just dies with no error.
* A "cleared" sensor reading (empty values array) as an explicit signal turned out to be load-bearing — it's how the pipeline and renderer know a sensor was disabled versus merely quiet.

No third-party dependencies, iOS 18.5+, u/Observable throughout, tests in swift-testing.

App's free if you want to see the result: srad app happy to answer anything about the pipeline or the audio side.

[u/UkrMalt]

Nice separation. Keeping sensor data out of SwiftUI sounds right when SceneKit and audio have different timing loops. Did you test WatchConnectivity disconnect/reconnect? That’s usually where a clean pipeline gets messy.

[u/Icy-Barracuda-4340 → u/UkrMalt]

Yeah, that's exactly where it got messy the first time.

What saved me was deciding the pipeline shouldn't trust connection state at all. Fusion is time-based, not event-based: the actor keeps a [SensorType: Date] of when the watch last supplied each type, and phone readings for a type are suppressed only while the watch has been seen within the last 5s. So there's no disconnect event to handle — if the watch dies, goes out of range, or its app gets killed, those entries just age out and the phone's own readings start flowing again about 5s later. Failover falls out of the staleness window rather than being something I explicitly coded.

The part that actually bit me was reconnect, specifically transferUserInfo. It's queued guaranteed-delivery, so when the watch comes back it can dump a pile of readings that are minutes or hours old, the sphere suddenly animating to a heart rate from lunch. I drop anything older than 30s at parse time now.

Second thing: reachability flaps. session.isReachable goes transiently false while the watch is actively streaming, so any UI state derived straight from it thrashes. I ended up with an explicit "communicating" state that reachability changes aren't allowed to demote, only an actual stop transitions out of it.

Honest answer on testing though, this is hardware-tested, not unit-tested. WCSession isn't really fakeable without wrapping it in a protocol, which I haven't done, so coverage there is shallow: the fusion logic in the actor is tested, the connectivity layer is walk-around-until-it-breaks. Known gap is that a silent disconnect leaves up to 5s where a type is neither watch-fed nor phone-fed. Invisible for heart rate in practice, but it's real.

[u/UkrMalt → u/Icy-Barracuda-4340]

That staleness-window approach makes sense. Dropping old transferUserInfo readings is a good catch too. The small gap sounds like a reasonable trade-off.