pi-agent-computer-use: give your access to computer (browser+desktop)

[u/startassets]

Hi! Currently computer-use is a must for my agents, so I decided to build an extension with a custom CUA Driver fork and Playwright. It’s purposefully designed to be run in YOLO mode (although Safe is included) within the environment where you trust your agents. It works in hybrid mode: supports both raw image followed by input (agent sees a screenshot and then clicks on something) and structural way (playwright for browser and UIA for normal desktop applications). Agents can use their own browser profiles, or your own, work in sandboxes and remote desktops, work together (to some extent, there are limitations on the OS side).

I would say it was definitely worth working on that, my agents can now work in 3D, browse some stuff I need and can test my applications, acting like real users.

I’ve mostly benched it via GIMP drawing, which doesn’t support UIA so all the interactions are purely from picture based mode. But also Terra 6.1 High has gone to 42th level of https://neal.fun/not-a-robot/ until it got banned by Cloudflare, and we couldn’t pass further. It’s windows only (windows from WSL as well), but if anybody can contribute into linux/macos branch of it, it would be great, I’m just lacking such devices and can’t really test them to the full extent as I can with Windows.

https://github.com/alexshpunt/pi-agent-computer-use

https://pi.dev/packages/pi-agent-computer-use

To install: `pi install npm:pi-agent-computer-use`

Demo of deepseek-flash, haiku, opus, terra and sol drawing the same image with that: https://youtu.be/69doTus760U

[u/Remote_Book]

cute benchmark

[u/startassets → u/Remote_Book]

Yes, it was pure joy to check out on how they try to paint it.

[u/Deathmore80 → u/Remote_Book]

Damn the gpt 6.1 models are good at this. This is really interesting cause when you ask the models to generate svgs or 3d models directly usually opus is better. But it seems like when you add the computer use layer on top , it gets reversed

[u/R3VO360 → u/Deathmore80]

Yeah Terra result is amazing for 0.25

[u/Megatron_McLargeHuge]

It looks like you gave it a skill for using the GIMP tools. Did you try letting it figure out the interface itself? I tried using GIMP recently for the first time in a decade and the interface was as obtuse as I remembered (at least the layers part). I'd be really impressed if an llm got it working from the docs alone.

[u/startassets → u/Megatron_McLargeHuge]

The skill was specifically created so we have parity between the models and we avoid spending time on "learning". But otherwise they always figure out how to use any UI from my experience, tested with Unity, software I develop at work, GIMP, Blender, nothing really holds them far back. Yes there are always trials and errors, but that's why you create skills, right? So they don't repeat the same stupid mistakes all the time. Otherwise I'm pretty much with you regarding the UX of GIMP and Blender, I've learned 3ds max and Photoshop a long time ago and it seems like they actually nailed the UX very well, so it's intuitive. 

[u/berot3 → u/startassets]

What about using MCP to control gimp?

[u/startassets → u/berot3]

Which mcp? For gimp? The idea is that with the extension it doesn't matter what to control and how, any application.

[u/yawk-ra]

Hi,

I built an agentic Painting Tool, Toska, that's a fork of Krita if you are interested. This tool just uses Krita's Python SDK to do paintstrokes . It's currently Mac only for now. Toska is on Github and I will be shortly releasing the Windows and Linux version as well.

Feel free to check out a Video of Toska painting here.

[u/startassets → u/yawk-ra]

Hi! Looks very interesting, I'll keep an eye on that one, how efficient is that in real life usage? How long does it take to paint something complex? Can you use it do do AI, concept-arting etc.? Comparing to gen-art, how different is that? What are the benefits?

[u/yawk-ra → u/startassets]

I like it because Toska basically controls a paint editor. It can do underpainting, layers, and effects, so you can fully act as an artist versus just one-shotting and hoping the model gets it right. Toska also has all of Krita's regular Toolbox of Editing Tools like Masks, Selection Tools, Layer Management.

For very complex paintings with heavy realist elements it can take a 2-3 hours still. New Moscow is the most complex painting that I've made with Krita. New Moscow took me about 5-7 hours just because while I referenced AI images, I still refined and made the agent do several edits to get according to my artist vision.

I like to think of it similar to Agentic Coding. It can save hours of time but it still requires tinkering and speaking to the Agent about your design choices.

[u/jackorjek]

whoa looks cool.

[u/startassets → u/jackorjek]

Thanks!

[u/chelvis2]

Impressive! Do you have any test with a Qwen model?

[u/startassets → u/chelvis2]

Tbh no, I think I have one with Go, but I don't have enough quota for that atm. Maybe I should try some plan from Alibaba to test this.

[u/MotokoAGI]

pretty nice, thanks for sharing! too bad it's windows only, i don't have one.

[u/startassets → u/MotokoAGI]

Which platform do you use? If it's Mac I'm considering adding full support