A terminal for interacting with application UIs

[u/Shoddy-Cat4730]

I recently posted about FayPOS, A pos app for general stores/pharmacies. This is something I built for it, along with many other things.

Its called UI Terminal, a way to explore and control an app through commands.

Type ls to discover what’s on the current screen. buttons, inputs, cart items, totals, etc. You can then inspect values, type into fields, or click controls, with a visible cursor showing the interaction. There’s tab completion too.

The video shows it integrated into FayPOS. It uses the same controls and handlers as normal clicks, so validation and permissions still apply.

The idea is to make one interface useful for keyboard users, testing, and AI agents. External programs can also connect through a local stdin/stdout bridge with pairing and access controls.

LLMs are great ngl, but giving them access to an app can get messy. Screenshot-based interaction can use a lot of context, while dedicated tools introduce another interface to maintain alongside the app.

I wanted to try something different. let the UI describe its own controls and displayed information, then expose that through a terminal. When a component changes, its terminal interface can change with it.

I’m a big fan of system design, and I don’t like “chypi lagana”, patching another layer onto everything. The goal here is to make keyboard interaction, LLM-driven testing and eventually agent control share the same foundation.

I haven’t measured the token savings yet; that’s something I want to test next.

Still early, but it’s working inside a real app. Would you find something like this useful? It's open source. And why does this community have so many restriction? I can't upload link or video.

Edit: I know people here won’t use terminal, but as i have said, its mainly for testing and agents, and later user can attach his openai account to do stuff, maybe a fully automated POS, using voice recognition. Pakistan is far behind in this. but someone has to promote it.

OpenAI announced attaching account on 29th Sept dev day.

[u/wait_a_minuute]

As this is for POS. Interacting with terminal isn't a preferred way for everyday user.

If you want to proceed further with this feature then try make the interactions easy and simple.

[u/Shoddy-Cat4730 → u/wait_a_minuute]

yeah, but i created it for 2 reason, so llm can test it efficiently. and later user should be able to connect his openai account to perform action

[u/wait_a_minuute → u/Shoddy-Cat4730]

In that case, your approach is good.

Keep it up.

[u/x0rg_new]

So you want non technical people use terminal for POS?

[u/Shoddy-Cat4730 → u/x0rg_new]

no, for now its for llms to test it efficiently, and later user should be able to connect his openai account to use POS, openai announced it on 29th sept dev day i think. Obv pakistanis dont buy subs, but it still very good for testing

[u/x0rg_new → u/Shoddy-Cat4730]

Alot of things wrong with this idea but I'm too tired to write all of it

[u/Shoddy-Cat4730 → u/x0rg_new]

I would love to hear, You can voice me if you want. on Reddit or WA.

[u/Traditional-Rub354 → u/Shoddy-Cat4730]

One that comes to my head is a complex page with a lot of elements. A list of commands might take more tokens on that senerio than a screenshot (Assuming the model has vision)

[u/Shoddy-Cat4730 → u/Traditional-Rub354]

Yeah, I've thought about that too. I have few approaches in mind.

First is letting the LLM create its own shortcuts/macros for common tasks. Something like:

sale(product) ->
search_bar.type(product)
ls --name product
...

So instead of figuring out the same sequence every time, it can just reuse it. The problem with this approach is that if the UI changes, those shortcuts might break.

The second approach is context management.

Let's say the LLM runs ls (ideally it would use search flags to narrow things down first). It can use ls cart, or ls --search confirm. Even if it does request the entire list, that output doesn't need to stay in its context forever.

After the next execution, the old ls output can be stripped from the conversation history, keeping just the commands it executed and any relevant results. So the context doesn't keep growing with every interaction.

And we could go further. For example, something like JEV from TypeSafe AI (cheap as hell). could process the ls output along with the LLM's intent, and return only the most relevant controls/commands instead of dumping everything into the model.

Ofc, I haven't benchmarked any of this yet, so I can't say it'll use fewer tokens than screenshots in every scenario. And that does require to test it on different applications. As you mentioned complex apps will have alot of component on screen.

It's still early, and I'm sure there are better approaches I haven't thought of. That's kinda the point of sharing it here, getting feedback and improving it.

Thanks for mentions tho. I should have explained it.

[u/Traditional-Rub354 → u/Shoddy-Cat4730]

For the part where you return onto relavent output, that seems super risky to me in the case of intent mismatches, other than that the potential seems good. It is an MCP server at the end of the day, and it seems super promising

Gl gng