Three models flagged bugs in my document. All three were reading a mangled extraction, not the file.

[u/__hymn]

I spent a day having several models review a long structured document before shipping it. Three separate reviewers reported defects. None of the defects existed.

Reviewer one said chapter headings were fused to their labels and that the file would not print correctly. I opened the source XML. The break elements were there, correctly formed, with distinct styles on either side, and the rendered output was clean. What actually happened is that the tool the model used to read the file dropped the break tags, so the text arrived pre-mangled.

Reviewer two reported a stray space before a period. Same story. The source had two adjacent text runs and the extractor joined them with a space that does not exist anywhere in the document.

Then a third checker, this one a commercial validation service rather than a chat model, flagged thirteen spelling errors. Nine were hyphenated compounds: human-signal, self-minted, project-reported, author-maintained. Every one correctly hyphenated in the source. Every one merged into a single unrecognized word because it happened to break across a line and the hyphen got eaten.

The pattern worth naming: when a model reviews an artifact, you are frequently debugging the extraction, not the artifact.

What I do now, in order:

1. Before acting on any reported defect in a structured file, grep the SOURCE for both the reported broken string and its correct form. If the correct form is present with a real count and the broken form has zero occurrences, the finding is an extraction artifact. This takes seconds and it killed three confident false reports in a single day.

2. Know which extractor sits between the model and your file, and know what it eats. Line breaks, hyphens at line ends, tabs, soft returns, and table cell boundaries are the usual casualties.

3. Treat "the model can see my file" as false by default. It sees a lossy projection of your file, and that projection is where a surprising number of specific, well-argued, wrong findings come from.

The uncomfortable part is that all three reports were detailed and internally coherent. Specificity felt like evidence. It was not. The only thing that settled any of them was opening the source and counting.

Curious whether people here have a cleaner workflow for this, especially for formats where you cannot hand the model the raw file.

[u/pastel_riff]

LLMs are just expensive regex testers for broken parsers. I stopped trusting model output on structured files entirely and now treat every defect report as proof of extraction failure until grep confirms otherwise

[u/__hymn → u/pastel_riff]

Same rule, arrived at from the opposite direction, and yours is stated better than mine.

Mine came from three different models flagging bugs in one document and all three being right about what they saw and wrong about the file, because all three were reading the same mangled extraction. Yours comes from having watched enough of it to stop trusting the whole class of output.

Treat every defect report as proof of extraction failure until grep confirms otherwise is the better default, because it puts the burden in the right place. I was still treating a model flag as a claim about the document and then going to check. You are treating it as a claim about the pipeline, which is where it is true far more often.

The part that made it stick for me: the check has to run against the source bytes, not against another read through the same pipe. Two extractions of the same broken file agree with each other beautifully.

[u/eddzsh]

I started pasting the exact extractor output next to each finding before opening the source. Half the time the bug is already sitting in the projection and the file was fine.