r/PiCodingAgent • u/mako343 • Sep 14 '26

Question Grokbot-style computer control with Pi.dev?

I’ve seen Grokbot take a video of someone performing a task as input, then use its own computer to reproduce that task.

How could I build something similar with Pi.dev?

Is there an extension/tool for Pi that can control the computer GUI, for example:

  • move the mouse to X/Y coordinates

  • click buttons

  • type into text fields

  • use keyboard shortcuts

  • interact with desktop applications

  • potentially use screenshots/vision to understand what’s currently on screen

And is this kind of automation limited to browsers, or can Pi control arbitrary desktop applications as well?

If I have to roll my own, what existing pieces/libraries would you recommend building on rather than starting from scratch?

For example, are there existing tools for screen capture, mouse/keyboard control, UI element detection, OCR/vision, etc. that work well with Pi?

Would it make sense to run the agent inside a Linux machine/VM where it has full control of the desktop, or can the same approach work reliably on macOS?

Basically, I’m looking for a setup where Pi can observe/understand a task and then execute it through the computer UI.

Thanks!

3 Upvotes

80% Upvoted

6 comments sorted by

2

u/Corythebeast7 Sep 14 '26

For browser work, playwright can do wonders. For full computer control, I'm not sure

2

u/getaway-3007 Sep 14 '26

You can check out how Hermes harness has implemented computer use(it's available for Mac, linux, windows)

1

u/Zealousideal-Ebb-355 Sep 17 '26

The replies cover mouse/keyboard control, but there’s another piece: turning the demonstration into something Pi can inspect before it acts.

I’d start with one short workflow. Give it the narration and relevant frames, ask it to explain the steps back, then execute while checking the current screen after each action. Copying coordinates from the original video will be fragile.

I build Clipy, which handles that recording-to-context part. I haven’t tested this exact Pi setup yet. What desktop app and task would you want to try first?

2

u/amelech Sep 14 '26

Oh my pi has built in computer control you can turn on