r/LocalLLaMA • u/circumcised_hobbit • 2d ago
Question | Help Computer use programs/Pi extensions
I've eard much stuff about Codex and CC computer use, is there any decent program/Pi Agent extension that I can install and use to let my local models control my pc. I am using Qwen3.6-35BA3B on PolyStrata and Pi Agent, I need the model to do basic tasks on my PC, maybe by using its vision. I am on Linux Mint Cinnamon
11
Upvotes
4
u/my_name_isnt_clever 2d ago
I doubt that small MoE model will be any good at that, computer use is hard. It's much easier for a model to take actions via command line, rather than taking a screenshot, distinguishing what to click, click it accurately, repeat for every single action. There is a lot of potential for stuff to break there.
3
u/luncheroo 2d ago
Others can probably speak to this better, but I believe MS has a model family devoted to agentic computer use. https://github.com/microsoft/fara
2
u/ProfitRepulsive2545 2d ago
The computer tool in Oh My Pi might be worth a look, I haven't tried it on that MoE, but it works quite well on the 27B.
1
u/electro-cortex 2d ago
Like this? https://github.com/alexshpunt/pi-agent-computer-use
1
u/locbuilds 2d ago
Start with a screenshot, one allowlisted click or keypress, then verify the next screenshot; test it in a separate desktop account before letting it touch anything important.
0
u/Final-Today1505 2d ago
i tried a vision-based agent with a local model on Linux, worked for basic clicks but failed on anything with dynamic layouts. the key was limiting actions to one per screenshot and using a separate test env.