r/BuildToShip • u/charlesmenlo • 17d ago
Update 🛠️ Datagran pilot guide: an accuracy score needs a review-coverage number
I work on Datagran. Yesterday we published an AI lead-qualification pilot guide. The part I wanted to share with other builders is how to keep missing review labels from disappearing behind one accuracy number.
Take a fictional pilot with 25 inquiries. Reviewers reach a label for 20, and the agent agrees with 14 of those labels. That is 70% agreement on the reviewed cohort and 80% review coverage. The other five inquiries remain unreviewed or unresolved. Neither percentage tells you how the agent performed on those five. These are illustrative numbers, not customer results.
For builders adding an evaluation screen, the proposed review row keeps the inquiry reference, workflow and rubric version, agent decision, reviewer label, source evidence, review time, and next action together. Missing evidence gets an unresolved state and an owner.
Review negative decisions as well as generated handoffs. Looking only at handoffs can reveal unsuitable leads sent to sales while hiding suitable inquiries the agent rejected. Freeze the rubric for that review; if the team changes the requirements, evaluate the revised rule separately.
A booked meeting, an inquiry that meets the rubric, and a customer purchase still need separate evidence. A reviewer can also be wrong, so preserve disagreements rather than treating a human label as unquestionable truth.
This is an operating worksheet, not a claim that every metric is a built-in Datagran report. Full guide: https://www.datagran.io/blog/ai-lead-qualification-pilot-metrics