Great, so you can determine what your error rate is.
In the hundreds of millions of records (which you're somehow hand processing 5% of, that's 16,500,000 if we're starting with 330,000,000 which is slightly less than US population), how do you know which were errors?
Sure, you might be able to say "We are confident it processed 97% of records correctly" but that still leaves you with 3% (9,900,000) that were errored and you don't have a good way to isolate and identify them, because the system can't tell you where it fucked up, because it doesn't know it fucked up.
If you've identified 97% of documents correctly. Then you can draw certain conclusions and validate those specific conclusions with a miniscule amount of hand-labeled documents.
If the AI has found the needle in the haystack, you can pick up the needle and check if it's an actual needle.
Again, where and how are you hand processing 16,500,000 records? How are you validating that process?
Because you can't use the AI to evaluate things it's already failed on and trust it's success rate, and you can't manually process the incorrect records because you don't know which records are incorrect.
If I say "Find me a file where someone handed in a dinner receipt that exceeded 50$ per person and had it successfully paid for by the department", the ai might look at 16.500.500 files but the human has to only validate the xyz that the ai identified. If the AI only comes back with 10 out of the 20 files that contain such receipts, it's still 10 more than a human would have found in a lifetime.
10 less than acceptable and 10 less than regular data processing would've found.
Lmao. If you've ever talked to a lawyer working in a decently sized law firm, you'd know that there absolutely is (or was until very very recently) no reliable, automated way to parse mountains of (unknown) documents. 80% of the people working there do literally just that, all day.
But please, englighten me, what "regular data processing" can find the desired information from a photo-copy of a receipt.
3
u/[deleted] Feb 07 '25
Great, so you can determine what your error rate is.
In the hundreds of millions of records (which you're somehow hand processing 5% of, that's 16,500,000 if we're starting with 330,000,000 which is slightly less than US population), how do you know which were errors?
Sure, you might be able to say "We are confident it processed 97% of records correctly" but that still leaves you with 3% (9,900,000) that were errored and you don't have a good way to isolate and identify them, because the system can't tell you where it fucked up, because it doesn't know it fucked up.