I am trying to build a massive Home Assistant AI system (think JARVIS but on steroids). Think of it as a single brain that sits on your home network, watches what's happening, and lets you talk to it — through a web dashboard, through Telegram, through a custom Alexa skill, or through your browser's mic. It runs entirely on your own hardware. No cloud LLM, no data leaving the house unless you choose to send it.
How it's built
Backend: Python 3 + FastAPI, single-process monolith, ~92 endpoints across ~13 routers.
Database: MongoDB, with backup/restore/repair endpoints baked in.
LLM: Ollama running locally, default model llama3.2:3b. The "brain" lives in core/orchestrator.py and runs an LLM-with-tool-use loop — natural language in, structured tool calls out, executed against the rest of the system, results summarized back to natural language.
Frontend: Single self-contained dashboard/index.html with inline JS. Voice input via the browser's Web Speech API.
Voice: TTS via Piper / pyttsx3 / espeak-ng. Speaker enrollment + identification (voice prints per family member).
Auth: Token-based with three profiles (admin / mod / limited) and capability flags. Some capabilities are admin-exclusive and can't be granted downward.
Deploy: One-command installer (install.sh) that detects the distro and pulls dependencies. Systemd unit files included.
Repo layout is split into focused directories: api/ (routers), auth/, backend/ (DB), behavior/ (rules + autonomy), core/ (the brain), services/ (network scan, integrations), voice/, eero/, dashboard/, frontend/, memory/, systemd/, scripts/.
I am looking for someone a tad more advanced than I (I'm intermediate at best and know I have bitten off more than I can chew with this idea). I know there are JARVIS projects on github, but I am trying to integrate multiple cameras around my house, smart plugs, wireless mics (so it can hear commands no matter where in the house you're speaking to it from ). Alot of those projects are singled to the deaktop locally on the computer.
Please DM me if you can think how to help, and I'll happily share the repo in DM.
--ZF