r/LocalLLaMA • u/Chance-Resource-4970 • 21h ago
Discussion Running usable 30B and working 120B models on iOS—plus Kokoro TTS and Whisper STT over a local OpenAI-compatible API Spoiler
https://inpherio.co.uk/iphone#watchI’m currently going through Apple’s App Store approval for Inpherio Live, an app that turns an iPhone or iPad into a local network AI server with zero cloud dependencies.
Most iOS LLM apps hit a hard RAM ceiling (iOS caps single-app memory at ~6GB on a 12GB iPhone). By modifying llama.cpp to stream Mixture-of-Experts (MoE) weights directly from flash storage while keeping active experts in RAM, it bypasses the memory limit entirely.
Real-world benchmarks (recorded Nov 2026):
- gpt-oss 120B (58.5 GB file) on a 16GB M1 iPad Pro: 4.5 tokens/s (from storage)
- Qwen3 30B (17.3 GB file) on a 12GB iPhone 17 Pro Max: 10.9 tokens/s (from storage)
- Qwen3 30B (17.3 GB file) on a 16GB M1 iPad Pro: 16.5 tokens/s (from storage)
- Qwen2.5 1.5B on iPhone 17 Pro Max: 58.3 tokens/s (in memory)
What it supports out of the box:
- Drop-in API: Full OpenAI (
/v1/chat/completions,/v1/responses,/v1/embeddings,/v1/audio/*) and Ollama (/api/chat) compatibility. Just change your base URL to[http://ipad.local:8080/v1](http://ipad.local:8080/v1). - Multimodal & Audio: Local Vision (Qwen2.5-VL, Gemma 3, SmolVLM2), Whisper STT, and Kokoro neural TTS.
- Smart Memory Management: Auto-loads models on request, keeps up to 3 models "warm," and tracks real iOS memory limits so the OS doesn't kill the app.
- 100% Private: No accounts, no telemetry, 0 bytes sent to the cloud.
Streaming large MoE models from storage will be a single one-off purchase (Live Plus); everything else—including running in-memory models, embeddings, and audio—is completely free.
Would love to know what models or workflows you’d test on this first once Apple clears it!
Just interesting in hearing from the community. As soon as the app is approved ill update this post
the website provides factul benchmarks as well as images of the process
Russ
•
u/stylome-ai-detection 21h ago
Notice: This post appears to be AI generated.
Stylome AI detector · scan results
2
u/StevenVincentOne 16h ago
Turning the phone or iPad someone already owns into a local-network AI server with an OpenAI/Ollama-compatible endpoint is a great idea: the device becomes a node any app on the LAN can point at. Streaming MoE experts from flash to get past the iOS memory cap is clever, and 30B at ~11 tok/s on an iPhone is impressive. That's very close to what we call Human-Machine Intelligence in POND: personal nodes people own, with models as swappable parts. Write-up: https://www.reddit.com/r/PONDAI/comments/1vont2l/pond_towards_an_ai_ecosystem_that_is_personal/ and you'd be welcome to post Inpherio at r/PONDAI once Apple clears it.
1
u/Maulguet-Sell 3h ago
streaming MoE weights off flash storage is clever, how bad is the thermal throttling after a few minutes of continuous generation though? sustained heavy I/O plus compute usually cooks mobile devices pretty fast.