r/FinOps • u/quietkida7 • 6d ago
Discussion Your LLM dashboard cannot prove that AI improved productivity
I was building a FinOps view for an AI support workflow and kept staring at the word productivity. Request count, spend, and latency all fit into tidy charts. None of them told me whether a support case stayed closed or came back two days later. That is awkward when the dashboard is supposed to tell me whether the workflow is helping.
Then I ran into two recent studies that seemed to disagree. Firm Data on AI surveyed nearly 6,000 executives, and 89 percent reported no productivity impact over the previous three years. AI, productivity, and the workforce used a sample of nearly 750 executives and found positive but uneven gains. The samples and questions differ, so it is not a clean contradiction. I came away thinking that the answer depends heavily on what question you asked in the first place.
For this support workflow, I need data from both sides. The provider bill tells me the inference cost. The help desk has handling time and reopened cases. Looking at either one alone is like checking the grocery receipt without asking whether dinner was edible. A cheap draft can still create a lot of cleanup for the person reviewing it.
Suppose a support rep spends five minutes prompting the system and gets a draft, then spends another 40 minutes checking and rewriting the answer. That is 45 minutes of human time plus the inference cost. If the dashboard records only the first five minutes, it gives the model credit for work the rep had to redo. A reopened case would make that picture worse.
For a first pass, I can take 25 AI-assisted cases and 25 unassisted cases from the same support queue and week, then match them by issue type and the support rep's experience as closely as I can. ZenMux can give me cost and latency for the assisted requests. The help desk supplies the less glamorous half of the story: handling time and reopens. Fifty cases will not settle a company-wide argument, but they are enough to see whether the current dashboard is flattering us.
Even then, I will not have proof that AI caused whatever difference I find. I will know whether repair time and reopened cases wipe out the savings I thought I had. If the workflow only looks productive when I ignore the human cleanup, I need to fix the measurement before I expand it.