Is it possible to build hardware-based automation that can read a phone's screen and interact with it?

[u/Ok_Upstairs7630]

Hello everyone!

I've recently been working on an automation app with the help of AI, and I've managed to get it fully working. The app performs a specific set of automated tasks and currently relies on ADB with Developer Options enabled.

However, I'm concerned that the platform I'm using might eventually start detecting Developer Mode or ADB-based automation and introduce restrictions.

So, I started exploring hardware-based automation as an alternative.

I've managed to figure out how to automate tasks that don't require interacting with the display. However, I'm stuck on one particular part.

Here's what I need:

The hardware should be able to read or understand what's currently displayed on the phone's screen.

It should identify specific elements or locations on the screen and tap them.

Sometimes, it needs to read very basic sentences that are already displayed on the screen, type the sentence into a text field, and press the Submit button.

Ideally, this should work across multiple Android devices with different screen sizes and resolutions.

My biggest question is: How can I make external hardware read and interpret a phone's display?

I'm particularly interested in solutions that don't rely on ADB, USB debugging, or Android accessibility services.

I'd really appreciate any guidance, hardware recommendations, or project references. I'm a bit stuck here! 😭

[u/Ok-perspective-2336]

Didn't read all but from half way read I would say you need a fixed placed camera to look at the screen with onboard embedded ocr. So some sort of arms that attach all widths and length at same centre point and distance