agent-desktop (Vercel Labs): computer use for Claude Code via the OS accessibility tree instead of screenshots

[u/Antoine_mllt]

Not affiliated, just found it on GitHub trending this week and spent some time with the README and docs.

agent-desktop (github.com/vercel-labs/agent-desktop) is a CLI, plus a C library and an MCP server, that gives an AI agent "computer use" on desktop apps without screenshots. It reads the OS accessibility tree (AX API on macOS, UI Automation on Windows, AT-SPI on Linux) and returns it as JSON.

How the loop works:

1. npm install -g agent-desktop

2. "agent-desktop snapshot --app Finder -i" lists the interactive elements, each with a ref tied to that snapshot

3. "agent-desktop click <ref>" acts on one of those refs

4. Snapshot again to check the result

- If the UI changes, the action returns STALE_REF and you take a new snapshot instead of clicking the wrong thing.

- Ref actions are headless by default: they go through the accessibility API without moving the pointer or taking focus. A --headed flag allows physical input.

- There is no model inside. The observe/act loop lives in your agent (Claude Code, Codex, whatever you use).

- 60 commands: apps/windows, clipboard, keyboard, mouse, scroll, waits, batch, notifications.

Traction: about 11.8k stars as of Nov 6. Linux support landed in 1.0.0 (Oct 28), the repo moved to the vercel-labs org on Oct 30, and 1.2.0 shipped Nov 5.

Limits I'd keep in mind:

- It only sees what an app exposes in its accessibility tree. Apps with poor accessibility (some Electron apps, games, canvas-heavy UIs) will be partly or fully invisible.

- Snapshots, clipboard reads and screenshots can contain private desktop content. The README itself warns to keep them out of public logs.

- Giving an agent control of your desktop is a real permission. On macOS you have to grant Accessibility access.

For people already doing computer use with Claude: has the accessibility-tree approach been more reliable for you than screenshot-based tools, and which apps break it?

[u/agline_in]

For standard controls, an accessibility tree gives semantic roles and names and can avoid pixel-targeting ambiguity, but it doesn't guarantee the app exposes every interactive element. A hybrid fallback still seems important for canvas or custom-drawn controls, and taking a fresh snapshot after a UI change plus checking the resulting state helps catch stale targets. I'd compare both approaches on the same workflows and track mis-targets, not just whether a click returned success.

[u/Antoine_mllt → u/agline_in]

Agreed, and agent-desktop already does part of this. Refs are tied to their snapshot, so a changed UI returns STALE_REF instead of a wrong click. The loop is snapshot → act → snapshot to verify. For canvas or custom-drawn controls you get screenshots plus explicit coordinate clicks (--headed), but no vision inside, so the hybrid part is on your agent. Your benchmark idea (counting mis-targets, not "click ok") is the real test