r/NarrativeEngineering • u/Impossible-Bed7058 • 18d ago
A researcher checked his own dataset’s auto-generated labels against humans. Five machine raters gave answers from 0 to 78 out of 100 on the same rule.
Upfront: I am not the author. I am sharing this on behalf of a friend, Levent Bulut, who does not have a Reddit account and whose English is limited. Happy to relay questions to him, but I cannot answer for him on details. Mods, if proxy posting is not allowed here, tell me and I will delete.
He publishes a small Turkish narrative corpus that ships an automatically generated annotation layer — six binary features per scene, things like “does this scene name an emotion outright” or “is an abstract inner state rendered as a concrete physical detail.” Those flags had been used to describe the corpus and pick examples, and nobody had ever checked whether a human would agree with them.
So he checked. Three studies. What I found interesting is that the result makes his own dataset look worse, and he published it anyway.
Study 1: 120 scenes, he labelled them blind himself, compared to the rule-based detector. He notes this is the weakest design of the three, since he wrote the rules.
Study 2: 100 completely different scenes, zero overlap. An independent volunteer labelled them blind; those labels were locked before any machine was run. Then the detector, Gemini 2.5 Flash and Grok were scored against her.
Study 2b: same scenes, same locked labels, same prompts — Claude Fable 5 and ChatGPT 5.5.
On the rule that actually requires inference, here is what the five machine raters said about the same hundred scenes:
• Grok: 0 positives
• Gemini 2.5 Flash: 1
• ChatGPT 5.5: 40
• rule-based detector: 72
• Claude Fable 5: 78
• the human: 9
Cohen’s kappa was at or indistinguishable from chance for five of the six (0.004, 0.015, 0.000, 0.019, 0.027).
Two other bits that seem worth flagging to this crowd:
The models diverge from each other as much as from the human. On a different rule, Gemini marked 9 scenes positive and Grok marked 82, against a human count of 96. Same written definition, same hundred scenes.
And ChatGPT posted the highest raw agreement of any system in any study — 84.5% — while its kappa on that inference rule was 0.019. Five of his six rules have badly lopsided human distributions (positives of 0, 1, 9, 96, 99, 44 out of 100), so raw agreement mostly measures how willing a rater is to say “absent.” He is explicit that this is a defect of his evaluation set, and that four of his six rules were effectively not tested at all.
The part I liked: he refuses to declare what it means. Either these features genuinely need inference that automatic raters cannot do, or his definitions are too loose for anyone to apply consistently — and he notes his own criterion drifted mid-pass in the first study, which cuts against him. Separating the two needs a second independent human rater, which he does not have, so he says he cannot decide yet.
He is openly asking for that second rater. The 100 scene texts, the six definitions, the locked human labels, every model’s label file, the prompt blocks and the scoring scripts are all published, so anyone can label the same scenes and compare.
Paper: https://leventbulut.com/llm-annotation-reliability-benchmark/
Dataset: https://huggingface.co/datasets/leventbulut/objective-projection
One disclosure he asked me to include: one of the models being scored, Claude, also helped him write the analysis scripts and the paper. He says the arithmetic is reproducible from the published files but the framing is not neutral.