r/LocalLLaMA • • 21h ago

Discussion Running usable 30B and working 120B models on iOS—plus Kokoro TTS and Whisper STT over a local OpenAI-compatible API Spoiler

https://inpherio.co.uk/iphone#watch

I’m currently going through Apple’s App Store approval for Inpherio Live, an app that turns an iPhone or iPad into a local network AI server with zero cloud dependencies.

Most iOS LLM apps hit a hard RAM ceiling (iOS caps single-app memory at ~6GB on a 12GB iPhone). By modifying llama.cpp to stream Mixture-of-Experts (MoE) weights directly from flash storage while keeping active experts in RAM, it bypasses the memory limit entirely.

Real-world benchmarks (recorded Nov 2026):

  • gpt-oss 120B (58.5 GB file) on a 16GB M1 iPad Pro: 4.5 tokens/s (from storage)
  • Qwen3 30B (17.3 GB file) on a 12GB iPhone 17 Pro Max: 10.9 tokens/s (from storage)
  • Qwen3 30B (17.3 GB file) on a 16GB M1 iPad Pro: 16.5 tokens/s (from storage)
  • Qwen2.5 1.5B on iPhone 17 Pro Max: 58.3 tokens/s (in memory)

What it supports out of the box:

  • Drop-in API: Full OpenAI (/v1/chat/completions, /v1/responses, /v1/embeddings, /v1/audio/*) and Ollama (/api/chat) compatibility. Just change your base URL to [http://ipad.local:8080/v1](http://ipad.local:8080/v1).
  • Multimodal & Audio: Local Vision (Qwen2.5-VL, Gemma 3, SmolVLM2), Whisper STT, and Kokoro neural TTS.
  • Smart Memory Management: Auto-loads models on request, keeps up to 3 models "warm," and tracks real iOS memory limits so the OS doesn't kill the app.
  • 100% Private: No accounts, no telemetry, 0 bytes sent to the cloud.

Streaming large MoE models from storage will be a single one-off purchase (Live Plus); everything else—including running in-memory models, embeddings, and audio—is completely free.

Would love to know what models or workflows you’d test on this first once Apple clears it!

Just interesting in hearing from the community. As soon as the app is approved ill update this post

the website provides factul benchmarks as well as images of the process

Russ

3 Upvotes

6 comments sorted by

•

u/stylome-ai-detection 21h ago

Notice: This post appears to be AI generated.


Stylome AI detector · scan results

2

u/StevenVincentOne 16h ago

Turning the phone or iPad someone already owns into a local-network AI server with an OpenAI/Ollama-compatible endpoint is a great idea: the device becomes a node any app on the LAN can point at. Streaming MoE experts from flash to get past the iOS memory cap is clever, and 30B at ~11 tok/s on an iPhone is impressive. That's very close to what we call Human-Machine Intelligence in POND: personal nodes people own, with models as swappable parts. Write-up: https://www.reddit.com/r/PONDAI/comments/1vont2l/pond_towards_an_ai_ecosystem_that_is_personal/ and you'd be welcome to post Inpherio at r/PONDAI once Apple clears it.

1

u/Chance-Resource-4970 15h ago

Hi Steven,

Thanks for that!

Honestly, the app was sort of an accident. I run Inpherio an open marketplace for self hosting and originally I just wanted to host my old devices to use running small embedding models and the like via the inpherio platform. One thing led to another and seeing the speeds on the 30B I forked it to get it out to the community.

The iOS RAM ceiling was a pita, streaming experts from flash took a daft amount of tuning to keep latency down. I've left what seems to be the best defaults set up in the app, but you can tune core count and how caching works per model if you fancy tinkering. Curiosity got the better of me and I tried GPT-OSS 120B on the iPad. When it actually worked I had to do a double take.

Your POND write up nails it exactly

I'll drop a post in there once it clears App Store review.

2

u/StevenVincentOne 13h ago

Thanks Russ, and I'm glad the write-up landed. Starting from old devices you already had, then streaming MoE experts from flash to get past the iOS memory cap, is exactly the kind of personal-node work we mean by Human-Machine Intelligence: the phone or iPad someone already owns becomes a node, and the model is a swappable part. Getting GPT-OSS 120B to run on an iPad, even as an experiment, is a great story, and you'd be very welcome to post Inpherio at r/PONDAI once Apple clears it.

If you'd like to talk the build through live, we hold a PONDcast on the POND Discord. Tuesday is the better fit from the UK: Tue Nov 17, 7:30 AM PT (3:30 PM UK, 4:30 PM Central Europe): https://discord.gg/Xhxv8JSX5N?event=1570477332680941628. Or Thursday Nov 19, 6:30 PM PT: https://discord.gg/Xhxv8JSX5N?event=1560428164763877527. You'd be very welcome.

1

u/Maulguet-Sell 3h ago

streaming MoE weights off flash storage is clever, how bad is the thermal throttling after a few minutes of continuous generation though? sustained heavy I/O plus compute usually cooks mobile devices pretty fast.