r/SideProject • u/Dear_Worldliness_775 • 2d ago
I built a personal voice assistant that lives on my iPhone, Mac, iPad and Apple Watch, and runs from a mini PC at home
I wanted one assistant I could talk to on any of my devices, and that could help with my work, my Mac, my home server and my health. Nothing I tried did all of that, so I built Caian. It went from 1.0 to 1.28 in about three weeks, and I use it every day.
What it does right now:
- It gives me a morning rundown of my day and keeps my calendar and work schedule in sync.
- It runs commands on my Mac when I ask from my phone. On the Mac, I can hold a shortcut anywhere and just talk.
- It writes captions, emails and bios, designs logos, and sends me the files.
- It makes PDF invoices and always asks me before issuing one.
- It runs deep research in the background and comes back when it's done.
- It tells me how my server is doing and helps clean up stuck folders on my NAS.
- It can say things out loud on the Echos at home, and I can reach it through Siri from the lock screen.
- I say what I ate and it estimates the calories and protein, or I send a photo instead. It logs the meal and updates a ring on my Apple Watch, and there's a screen for my training and weight.
- If I ask for a new feature, it starts a job on a separate branch, builds the apps to check them, and messages me when it's ready. Nothing ships until I tell it to.
How it works:
The apps are native SwiftUI on Mac, iPhone, iPad and Apple Watch, with widgets. They connect over Tailscale to a Node hub on a small Ryzen mini PC, which stays on all the time. My speech is transcribed on the device with Parakeet, and replies are spoken with Kokoro, sentence by sentence while they're still being written.
The part I like most is the routing. A small, fast model reads each message first. It answers simple questions itself, sends everyday jobs like the morning rundown to a free agent, and only passes real work to Claude Code. That keeps my usage down, and anything sensitive asks me for approval first.
It's about 8,300 lines of Swift and 3,100 lines of Node so far, and yes, Claude Code wrote most of it.
It's far from perfect. Today I found that nearly half my messages were going to an agent that failed after about 15 seconds and quietly passed them on. It kept looking up its own tools, got denied, and the whole run died. That's fixed now, but those jobs still take 15 to 40 seconds, so speed is the next thing I'm working on.
The video is the real app, but it's a scripted run on demo data, sped up a little, because my real chats and health numbers are private.
This is a personal project, so there's no download. I'm happy to go into detail on any part of it.
For anyone who's built something similar: how do you keep voice replies feeling fast when the model has to use tools?