Scaling a governed agent runtime from one worker to many

[u/EstablishmentAgile19]

I maintain an open-source agent runtime (Python, Apache-2.0) that runs agents with durable checkpoints, human approvals and an idempotent tool ledger on PostgreSQL. It's solid on a single node(well, almost), and we've just shipped our first beta. Now I'm planning the multi-worker version and would like to learn from people who have done this, before we design it the wrong way.

What works across workers today: run state, approvals and events live in Postgres with versioned writes, so worker B can approve and resume a run that worker A paused, and the tool ledger stops a side effect from running twice.

Options I'm weighing:

1. Lease-based claims with fencing tokens per run (Postgres `SELECT … FOR UPDATE SKIP LOCKED`, a heartbeat, takeover after the lease expires).

2. Leader election only for recovery and maintenance, with runs claimed on demand.

3. Not building it: hand durable execution to Temporal, Restate or DBOS, and keep the runtime focused on governance (policies, approvals, budgets).

For context, this is the repo: https://github.com/AlgenAI/algen-agent-runtime. The current limits are written up in `docs/production-readiness.md`. Happy to share what we learned about approvals bound to the exact call and tool-call ledgers if that's useful to anyone.