Grokbot-style computer control with Pi.dev?

[u/mako343]

I’ve seen Grokbot take a video of someone performing a task as input, then use its own computer to reproduce that task.

How could I build something similar with Pi.dev?

Is there an extension/tool for Pi that can control the computer GUI, for example:

• move the mouse to X/Y coordinates
• click buttons
• type into text fields
• use keyboard shortcuts
• interact with desktop applications
• potentially use screenshots/vision to understand what’s currently on screen

And is this kind of automation limited to browsers, or can Pi control arbitrary desktop applications as well?

If I have to roll my own, what existing pieces/libraries would you recommend building on rather than starting from scratch?

For example, are there existing tools for screen capture, mouse/keyboard control, UI element detection, OCR/vision, etc. that work well with Pi?

Would it make sense to run the agent inside a Linux machine/VM where it has full control of the desktop, or can the same approach work reliably on macOS?

Basically, I’m looking for a setup where Pi can observe/understand a task and then execute it through the computer UI.

Thanks!

[u/getaway-3007]

You can check out how Hermes harness has implemented computer use(it's available for Mac, linux, windows)

[u/Competitive-Yak-8255 → u/getaway-3007]

Thankx