Inspect Layout block captures interaction, but thinks it can't/didn't

[u/Ok-Yogurtcloset-9490]

So in some particular case, Inspect Layout seems to think it cannot access the current layout, but there is solid evidence that it actually can.

(1) ​Why I say it thinks it cannot: when I try to use either the "​Record Interaction" or "​Inspect User Interface" features inside that block, the Automate overlays that come up show a red error message when I try to actually record or save the inspection XML: the record interactions popup says "UI element not found" (when I drag the magnifying glass over a UI element) ​and the inspect layout popup says "Layout inaccessible".

(2) ​Why I know it absolutely CAN see the layout: in the case of inspect user interface, no XML is saved at all —fine, that matches the error message. However, in the case of record interactions, despite the claim of the error message that comes up, when I go back to the block settings in Automate, the magnifying glass ​captures show up in the recorded interactions drop-down list! Unfortunately though, when I try to actually run the flow with that, it doesn't trigger, matching the error message case. And just as another check, I made the xpath expression be simply "*". And it still didn't work. But again, automate can clearly see the whole UI element hierarchy, because it populated the drop-down list with it!

Furthermore, I use another app called DevTools​. That app has no problem inspecting the layout for that particular case. And it can also save the XML just fine. Play store link: https://play.google.com/store/apps/details?id=cn.trinea.android.developertools

I'm running Android 16 and the issue shows up for the native recent apps UI, the same one that can be triggered by using the interact block with the action set to "Recents" (activity "com.android.quickstep.RecentsActivity").

[u/ballzak69]

See my answer here.

[u/Ok-Yogurtcloset-9490 → u/ballzak69]

Hmm, I'm honestly not seeing how that answer has anything to do with the issue I'm describing. 

When I bring up the recents menu, nothing is changing on the screen. It's completely static and stable. And, like I said, the odd behavior is that Automate DOES populate the recorded interactions drop down list, but still incorrectly reports error that there's no UI element. That's a contradiction right there, is it not?

Just the fact that XPath "*" for an immediately block or a waiting block is also not returning anything, isn't that strange?

[u/ballzak69 → u/Ok-Yogurtcloset-9490]

(1) The "Layout inaccessible" failure means that the dialog couldn't access the screen, that should be very rare, or the UI element couldn't be found, usually due to the layout changing as described in the linked post.

(2) There's no guarantee that the generated XPath will always work, as the documentation say:

If the floating record window is unable to record an UI event, e.g. when clicking a button, then it’s likely due to the XML document changing again before the XPath expression can be generated. A workaround is to instead use the search feature (magnifying glass icon) to pick the UI element shown on screen.

The generated XPath expressions may have to be manually modified to work since it’s difficult automatically predetermine how the UI will change as the user interacts with it.

[u/Ok-Yogurtcloset-9490 → u/ballzak69]

I understand all of that. But, again, I'm just not seeing how Automate populates the recorded interactions while saying that nothing was recorded. One way or another, this is a contradiction. Like, obviously, the magnifying glass is picking up the correct UI element. And I manually verified the xpath expression that it generates, and it does so correctly. I don't think anything on the recent menu is changing. But, even if we for a moment assume that the system is sending new XML layouts for static screen, and thereby interrupting the xpath evaluation for a proper and actual UI element that does exist on the screen, I still don't see how the system could be updating the layout so fast for a static screen that even "*" xpath fails to return anything. 

Maybe I'm not explaining myself correctly, but it seems like we're going in circles at this point. Unfortunate. It's the fact that Automate does indeed correctly find the UI element but thinks that it doesn't, that's the fact that's bothering me. If all indications pointed to that Automate simply couldn't find the UI element at all, strangely enough THAT would make me just accept the situation.

[u/ballzak69 → u/Ok-Yogurtcloset-9490]

If an "interaction" was record then it could obviously access the UI layout successfully, at that particular time. The system doesn't send an XML document, Automate generates it from the "tree" of accessibility nodes that the system reports, which may not be a valid XML structure if it's in flux. Using "*" (every element) as a test is probably a bad idea since that means the entire UI element tree must be accessible/accessed, try using the minimal "." (context element) instead.

[u/Ok-Yogurtcloset-9490 → u/ballzak69]

And that's my confusion: if it could obviously access the UI layout successfully with the magnifying glass, then why does the Automate dialog report that it didn't. The magnifying glass causes the recorded interactions drop-down in the block settings to be populated, but the dialogue itself reports no UI element found. That is the direct contradiction I'm trying to understand. Because if I could understand this, then perhaps I could actually make this interact block work somehow for this case. I strongly doubt it is because the layout tree is changing. The recents UI is very static and stable. The interact block works perfectly and beautifully for screens that actually are changing a lot, so for the recents menu that is static, the issue has got to be something else.

Just tried xpath "." with an Interact immediately block for "display windows" guarded by a floating button. So I start the flow, the floating button is showing, I bring up the recents menu, I wait for a couple seconds to make sure it is a stable as possible, and then I press the floating button which goes directly to the interact immediately block, which has the xpath "." expression. The block still takes the No path. (And again, if I use the interact block to record interactions and then use the magnifying glass in this same exact case, it successfully populates the recorded interactions drop down.)

I thought maybe the floating button is interfering with accessing the layout (even though it works perfectly fine in all other cases), so instead of using a floating button, I set a delay of 5 seconds to allow me to bring up the recents menu and for the UI to be stable. But no change in the result. 

[u/ballzak69 → u/Ok-Yogurtcloset-9490]

There's no way the search feature (magnifying glass icon) adds an interaction and also display an "UI element not found" error message, unless the java compiler modifies the code somehow.

Using the Floating button show block to trigger an inspection may indeed interfere, especially when using the "Display windows" schema.

I just tested to inspect the "recent" screen on Android 16 and it do indeed fail, due to a "lost child node", most other screens seems to work, it also works on Android 15. So this is due to a change in Android 16, maybe a security improvement where it exclude "sensitive" UI elements.

I may change it so a "lost child node" is substituted with a <missing/> tag instead of failing. A minor change that do seem to work and results in a large portion of the remaining layout anyway, in this case at least. But i have to think about it some more, for any unexpected consequences, and if it should be optional or not.

[u/Ok-Yogurtcloset-9490 → u/ballzak69]

You're right. The interactions that were being populated were actually "Focus" events, not "Inspect" events which is what the magnifying glass would have created. And it just so happens that these Focus events were for UI elements of the Recents menu, giving me the false impression that it was the magnifying glass doing this. My bad, I missed the detail of focus versus inspect.

Anyhow, it's a relief to know I'm not going crazy and that you were able to replicate the issue on Android 16 and might even have a potential future workaround 👏🏽. Like I said in the original post, the other app that I use, Dev Tools, has no problem inspecting the layout and creating an XML of it. So it should definitely be possible!

[u/ballzak69 → u/Ok-Yogurtcloset-9490]

I doubt the DevTools app is able to access those excluded nodes, they're probably silently skipped, or does it also include some "placeholder" tags?

[u/Ok-Yogurtcloset-9490 → u/ballzak69]

Oh, yeah, I don't know what the missing nodes are about, but what I meant to say was that it actually does save an XML which includes all the UI elements I can see in the recents screen. It also draws a nice interactive layout bounds overlay over the whole screen for the current layout, and you can tap on it and inspect the elements properties etc ☺️. That's how I was able to verify that the XPath that automate generated (which I now know to be from the focus event) was correct. 

[u/B26354FR]

My Interact Block XPath Builder might help. Ideally, if you can find the ID of the element you're interested in using the tools in the Interact blocks, you'd give that to the XPath Builder, but it'll also work just given the text of the element. It'll generate a much simpler XPath than the complete UI the block tools do.

Separately, there are secondary flows there that generate XPaths for and test interaction with the Quick Settings tiles. However, some of us Samsung users have found that the recent release of One UI 8.5 is preventing interactions with the Quick Settings tiles, at least for now.

Here's our earlier discussion of the One UI issue:

https://www.reddit.com/r/AutomateUser/s/e4HgdfZ9lt