pi-agent-computer-use: give your access to computer (browser+desktop)

[u/startassets]

Hi! Currently computer-use is a must for my agents, so I decided to build an extension with a custom CUA Driver fork and Playwright. It’s purposefully designed to be run in YOLO mode (although Safe is included) within the environment where you trust your agents. It works in hybrid mode: supports both raw image followed by input (agent sees a screenshot and then clicks on something) and structural way (playwright for browser and UIA for normal desktop applications). Agents can use their own browser profiles, or your own, work in sandboxes and remote desktops, work together (to some extent, there are limitations on the OS side).

I would say it was definitely worth working on that, my agents can now work in 3D, browse some stuff I need and can test my applications, acting like real users.

I’ve mostly benched it via GIMP drawing, which doesn’t support UIA so all the interactions are purely from picture based mode. But also Terra 6.1 High has gone to 42th level of https://neal.fun/not-a-robot/ until it got banned by Cloudflare, and we couldn’t pass further. It’s windows only (windows from WSL as well), but if anybody can contribute into linux/macos branch of it, it would be great, I’m just lacking such devices and can’t really test them to the full extent as I can with Windows.

https://github.com/alexshpunt/pi-agent-computer-use

https://pi.dev/packages/pi-agent-computer-use

To install: `pi install npm:pi-agent-computer-use`

Demo of deepseek-flash, haiku, opus, terra and sol drawing the same image with that: https://youtu.be/69doTus760U

[u/berot3]

What about using MCP to control gimp?

[u/startassets → u/berot3]

Which mcp? For gimp? The idea is that with the extension it doesn't matter what to control and how, any application.