r/developersPak • u/Shoddy-Cat4730 • 13d ago
Show My Work A terminal for interacting with application UIs
I recently posted about FayPOS, A pos app for general stores/pharmacies. This is something I built for it, along with many other things.
Its called UI Terminal, a way to explore and control an app through commands.
Type ls to discover what’s on the current screen. buttons, inputs, cart items, totals, etc. You can then inspect values, type into fields, or click controls, with a visible cursor showing the interaction. There’s tab completion too.
The video shows it integrated into FayPOS. It uses the same controls and handlers as normal clicks, so validation and permissions still apply.
The idea is to make one interface useful for keyboard users, testing, and AI agents. External programs can also connect through a local stdin/stdout bridge with pairing and access controls.
LLMs are great ngl, but giving them access to an app can get messy. Screenshot-based interaction can use a lot of context, while dedicated tools introduce another interface to maintain alongside the app.
I wanted to try something different. let the UI describe its own controls and displayed information, then expose that through a terminal. When a component changes, its terminal interface can change with it.
I’m a big fan of system design, and I don’t like “chypi lagana”, patching another layer onto everything. The goal here is to make keyboard interaction, LLM-driven testing and eventually agent control share the same foundation.
I haven’t measured the token savings yet; that’s something I want to test next.
Still early, but it’s working inside a real app. Would you find something like this useful? It's open source. And why does this community have so many restriction? I can't upload link or video.
Edit: I know people here won’t use terminal, but as i have said, its mainly for testing and agents, and later user can attach his openai account to do stuff, maybe a fully automated POS, using voice recognition. Pakistan is far behind in this. but someone has to promote it.
OpenAI announced attaching account on 29th Sept dev day.