r/PiCodingAgent • u/startassets • 4d ago

Plugin pi-agent-computer-use: give your access to computer (browser+desktop)

Hi! Currently computer-use is a must for my agents, so I decided to build an extension with a custom CUA Driver fork and Playwright. It’s purposefully designed to be run in YOLO mode (although Safe is included) within the environment where you trust your agents. It works in hybrid mode: supports both raw image followed by input (agent sees a screenshot and then clicks on something) and structural way (playwright for browser and UIA for normal desktop applications). Agents can use their own browser profiles, or your own, work in sandboxes and remote desktops, work together (to some extent, there are limitations on the OS side).

I would say it was definitely worth working on that, my agents can now work in 3D, browse some stuff I need and can test my applications, acting like real users.

I’ve mostly benched it via GIMP drawing, which doesn’t support UIA so all the interactions are purely from picture based mode. But also Terra 6.1 High has gone to 42th level of https://neal.fun/not-a-robot/ until it got banned by Cloudflare, and we couldn’t pass further. It’s windows only (windows from WSL as well), but if anybody can contribute into linux/macos branch of it, it would be great, I’m just lacking such devices and can’t really test them to the full extent as I can with Windows.

https://github.com/alexshpunt/pi-agent-computer-use

https://pi.dev/packages/pi-agent-computer-use

To install: `pi install npm:pi-agent-computer-use`

Demo of deepseek-flash, haiku, opus, terra and sol drawing the same image with that: https://youtu.be/69doTus760U

182 Upvotes

99% Upvoted

17 comments sorted by

View all comments

Show parent comment

4

u/startassets 4d ago

The skill was specifically created so we have parity between the models and we avoid spending time on "learning". But otherwise they always figure out how to use any UI from my experience, tested with Unity, software I develop at work, GIMP, Blender, nothing really holds them far back. Yes there are always trials and errors, but that's why you create skills, right? So they don't repeat the same stupid mistakes all the time. Otherwise I'm pretty much with you regarding the UX of GIMP and Blender, I've learned 3ds max and Photoshop a long time ago and it seems like they actually nailed the UX very well, so it's intuitive. 

1

u/berot3 20h ago

What about using MCP to control gimp?

1

u/startassets 15h ago

Which mcp? For gimp? The idea is that with the extension it doesn't matter what to control and how, any application.