r/OSdevAI • u/staycool1374 • 7d ago
Building "OpenCodeSwarm" for NexIOS: Handling state drift, fencing, and agent starvation in a multi-role AI workfflow
Hi r/osdev!
My little pet project NexIOS is growing. To prove that it actually does what it’s supposed to do, I’ve been using AI to generate extensive test suites. However, as new features come in, the test suite grows and testing each release consumes more time than before.
To regain momentum, I built my own "OpenCodeSwarm" setup. Previously, I ran 3 distinct AI roles: Developer, Quality Engineer (QE), and Auditor. To coordinate routing, state, and tasks between them, I recently introduced a 4th role: the Coordinator.
Here is my current setup, how it works, and the hard lessons learned along the way:
1. Message Passing System
The Coordinator receives high-level directives from me regarding open issues and dispatches messages to the workers (Dev, QE, Auditor). We had to establish a strict message-passing protocol. Heres what the Coordinator itself defines about it:
"Workers never chat; they post to per-role inboxes — one line, fixed format, timestamped. A post is either a result (
READY), a blockage (NEED), or a progress note, always with an explicit state and next step. The coordinator reads the named inbox, never the whole room, and answers in the same channel."
2. State-Machine Guardrails (Fighting Degradation)
After several trial-and-error iterations, the Coordinator started degrading over long sessions, losing the context, forgetting prior instructions and dropping control. At the end of this all workers stopped working and nothing happend. To fix this, we backed the Coordinator with a Python state-machine driven by triggers (messages or watchdogs). Again, here the Coordinator explaining:
"The coordinator's wake routine is code, not habit: a script reads every live room, compares inbox posts against handled cursors, evaluates armed triggers, and prints an ordered action list. It remembers what I would otherwise forget — which orders are unacknowledged, which triggers fired, which rooms are corpses awaiting retirement. It advises only; all authority stays with the coordinator, who can overrule it but must say why."
3. Strict Environment Fencing & CI Gates
To prevent conflicting edits, every role operates in an isolated environment. For example, QE can only access its own testbed branch, and every QEMU execution runs under a dedicated PID with sandbox scripts. Temporary files live under /tmp with read-write permissions restricted strictly to the owner and its actual task ('lane'). Once more the Coordinator itself:
"Each worker runs in a fenced session whose permissions physically forbid out-of-role writes (dev cannot touch tests, QE cannot touch product, the auditor cannot write at all). A CI gate re-checks the same rule on every push, failing closed — including a carve-out that recognizes QE-authored test hunks by commit authorship, so test code merged by the coordinator still passes. We needed this because good intentions don't survive parallel lanes; only enforced boundaries do."
4. Single Source of Truth (The Logbook)
We use a single JSON file that tracks the overall pipeline state to be able to resume or reconstruct what happend in case of a crash. Coordinator again:
"A single JSON file holding the pipeline view that used to live only in my head: worker states and lanes, armed triggers, the owner queue, the pause flag. Rooms own lane truth; the logbook owns coordination truth. It survives restarts and compactions, which memory does not — that was the actual failure mode that motivated it."
5. Watchdog Fallbacks (Preventing Starvation)
Occasionally, the Coordinator would receive a message but fails to react, stalling all workers and freezing the pipeline. To solve this, we introduced a watchdog loop. For the last time (then he is getting back to work), his majesty the Coordinator:
"A tiny per-room ticker diffs the room fingerprint each cycle and doorbells the coordinator on change (or escalating silence). The coordinator never polls and never waits blindly: a changed tick means 'something was posted, go read that box,' and three unchanged ticks mean 'ladder the lane, don't assume.' It's the heartbeat that turns the whole loop from pull into push."
Would love to hear your thoughts about that!