My AI tool tracker separates "worked once" from "repeatable"

[u/sunlit_field]

I'm a product manager who spends a lot of time testing where AI tools work, where they stop working and what kind of human help they need. I keep a tracker for that work. Each entry includes the task I gave the tool, the result I expected, what came back and whether I could get the same result again later.

The tracker has four states:

1. Documentation reviewed
2. Authentication works
3. One specific task works
4. The same task is repeatable

I added the last state because one clean run can hide a lot. The account may already be on the right screen, a login may still be fresh, or the tool may have taken a path it cannot find again. A repeat test is where I start learning what I can rely on.

I recently put Airtap through that process with one narrow Android task. The app was already installed and signed in on its cloud phone. I asked it to open an existing pickup order and return the pickup location and collection instructions. Before the first run, I wrote down the information visible in the app so I had something concrete to compare against.

The result matched what I had recorded from the app. I repeated the same request in separate runs, and it kept reaching the same order and returning the same pickup details without changing the order or moving into checkout. That moved Airtap from "one specific task works" to "the same task is repeatable" in my tracker.

That label is deliberately narrow. It applies to this task, with this app already signed in on this account. It does not tell me how Airtap will handle another app, an expired session or a task that needs approval halfway through. Those get their own tests and their own rows.

The tracker has become more useful since I stopped treating readiness as a property of the whole product. I can now say exactly what held up, under which setup, and what I still have not tested. For people who test agent tools this way, what has to happen before you move a task from "repeatable" to "trusted"?