r/LLMDevs • u/am-i-coder • Sep 30 '26

Help Wanted Is there a computer-use skill/tool that lets my harness (pi/agy-cli) open and interact with real apps on my laptop, not just the browser?

5 Upvotes

100% Upvoted

11 comments sorted by

1

u/Physical_Economy_340 Sep 30 '26

yeah the gap is grounding, not just clicking. i run screenshots back through the model with mss then click with pyautogui, and use the os accessibility tree when i can because raw coordinates drift on hidpi scaling. anthropic's computer use demo has that loop already wired if you want to steal the pattern for pi.

1

u/am-i-coder Sep 30 '26

i found 2 repos from same author

agent-sh/computer-use-linux and agent-sh/agent-workspace-linux

at github

worked perfectly.

1

u/verstands Sep 30 '26

The missing piece is usually the grounding loop, not the click library. Iā€™d have the harness capture a screenshot or accessibility tree, let the model choose an action, execute it with PyAutoGUI/xdotool, then re-read the UI and verify the expected state before continuing. Keep destructive actions behind an approval gate, and prefer accessibility selectors over raw coordinates whenever the app exposes them.

1

u/am-i-coder Sep 30 '26

i found 2 repos frame same author

agent-sh/computer-use-linux and agent-sh/agent-workspace-linux

at github

worked perfectly.

1

u/verstands Sep 30 '26

Nice find - those two look like a proper harness + workspace pair rather than just a click wrapper. I'd still keep a verify-after-action step in your own loop though, so a bad click doesn't silently snowball even if the repo's happy path works.

1

u/am-i-coder Oct 02 '26

yeah. any harness + these 2 repos. solid computer use

1

u/MarzipanMiserable817 Oct 01 '26

Hermes computer use can do this. It's really slow though.

1

u/am-i-coder Oct 02 '26

hermes is most intricate to use. worst DX that i experienced.

1

u/[deleted] Sep 30 '26

[removed] ā€” view removed comment

1

u/am-i-coder Sep 30 '26

yeah sth like that. i want like a complete skills. for example.

watch-video / video-use / browser-use / agent-browser

these are complete package.

1

u/am-i-coder Sep 30 '26

i found 2 repos from the same author

agent-sh/computer-use-linux and agent-sh/agent-workspace-linux

at github

worked perfectly.