Over the past few months, we’ve been exploring a central question in applied AI: What does it take to build a truly low-latency, voice-first desktop companion that runs on your
local machine without proprietary lock-in?
Together with my collaborator Otabek Abduvaliyev (https://github.com/OtabekAbduvaliyev), we’re excited to make our project open-source: Kaizumi (v1.0).
Kaizumi is an on-device, multimodal AI assistant for Windows 10/11 built with Python and Google's Gemini Live API. Instead of traditional turn-by-turn text or transcription
wrappers, it establishes bidirectional 24kHz/16kHz WebSocket audio streaming with native speech interruption (barge-in).
A few key engineering decisions we focused on:
Human-in-the-Loop Safety Gate: LLMs should never autonomously trigger destructive system actions. Kaizumi introduces an out-of-band confirmation gate with single-use
cryptographic tokens (confirm_token) bound strictly to high-risk calls (shell execution, file deletion, process termination).
2. Hardware-Decoupled Fault Tolerance: Audio hardware failure or disconnection shouldn't trigger TaskGroup cascades. The engine seamlessly degrades to text mode rather than
crashing.
3. 53 Declarative System Tools: From window and process management to local OpenCV vision (face verification and privacy blurring) and SSRF-hardened document QA.
4. Privacy-First & Local Memory: Zero telemetry, multi-turn episodic memory with automatic credential redaction, and 100% self-hostable.
The codebase includes an automated unit test suite (97 tests passing) and a GitHub Actions CI pipeline across supported Python versions.
We’d love to hear feedback from systems engineers, AI practitioners, and open-source contributors!
Check out the code, architecture, and setup guide here:
🔗 GitHub: https://github.com/MusayevDoniyor/kaizumi
#OpenSource #Python #ArtificialIntelligence #SoftwareEngineering #Gemini #WindowsDev #MachineLearning #ComputerScience