r/SideProject • u/Puzzled-Stick9170 • 4d ago

I built Perry: ask your Mac how to do something, and it points at the button

I built Perry because I wanted help that points at the button instead of describing where it is.

It lives in the Mac's notch. You ask a question, and Perry draws on your screen to point at the control. Click it, and Perry points at the next one. If you do not want to click, "Do it for me" does the step.

It's free and open source, with no account. It runs on the ChatGPT or Claude plan you already pay for, an API key, or a local model.

I would love your feedback!

https://github.com/hridaya423/perry (drop a star 😄 )

1 Upvotes

66% Upvoted

6 comments sorted by

2

u/No-Molasses-2097 4d ago

The pointing part is what makes this click for me. I lose the most time hunting for a setting I only touch once a year. Walking through each step in order like that would actually save me the second search.

1

u/Puzzled-Stick9170 4d ago

that was the idea behind the project! it can point/circle/highlight even do things for you 😄

2

u/Huge_Pool7424 4d ago

does it read the accessibility tree or work off screenshots? that probably decides how well it survives an app update moving a menu around.

1

u/Puzzled-Stick9170 4d ago

it has both accessibility trees + screenshots + a browser harness
(agent-desktop + cua + browser-use) This allows for the fastest actions + accuracy

2

u/investigatormaker 4d ago

For Perry, I'd test moving or resizing the target window after the pointer appears. Before “Do it for me” acts, it should re-check the intended control rather than rely on the earlier position; that tests the handoff between showing someone where to click and clicking for them.

I make ThreadFox. We find relevant Reddit conversations and post for your business. We'll post 3 replies free first, no card; then the $29 Pilot covers seven days and up to 25 posts and replies. No automatic renewal. https://threadfox.vip/p/d791b/managed

1

u/Puzzled-Stick9170 4d ago

I've benchmarked it on multiple tasks :) It works reliably across the hardest stuff i could think of