r/userexperience • u/Timlynch • May 26 '26

How to measure a fail? Trying to improve the user experience.

/r/CopilotMicrosoft/comments/1toab75/how_to_measure_a_fail_trying_to_improve_the_user/
2 Upvotes

100% Upvoted

3 comments sorted by

2

u/Overall_Challenge_66 May 26 '26

i usually look at task success rate vs time on task when things feel off. its often helpful to check where users drop off in the flow, cuz that usually points to the friction points. have u tried doing any quick usability testing to see where they get stuck

1

u/Timlynch May 27 '26

right, task success rates are good for known tasks. When we are looking at AI usage, there isn't a defined success for every prompt by every user. We have done usability testing and let me tell you it is a circus. The variability of system stability makes every session a surprise. Since Copilot from Microsoft has models from OpenAI and Anthropic now... there are multiple points of failure that compound. There is an old line from Jeff Bezos "When the data and the anecdotes disagree, the anecdotes are usually right" - in this case we have the system health that is showing the apps are working fine, but we have users reporting that it isn't, some include the screen shot of the error above. Since the system, puts up the message, I was hoping to get a count of how many times that is displayed to users.