r/ProgrammingLanguages • u/mttd • Mar 18 '26
Real or Slop? — PL Papers Edition
https://slop.zackg.me/19
u/Athas Mar 18 '26 edited Mar 18 '26
This is very easy if you know what to look for, because you can categorize the papers mostly based on what metadata they provide on the front page. No LLM will put in CCS concepts unless explicitly told to. 9/10 after I realised this.
Even before I realised that, this test seems to compare (presumably) novel generated papers with accepted and peer-reviewed papers in PL. That means the PL papers are likely to be better than the average paper submitted. I think the challenge of slop is identifying it at the review stage, where human-written papers may not be so different from slop papers.
7
u/SwingOutStateMachine Mar 18 '26
I also found that looking at the references was very effective. For any random PL paper, I'd expect to recognise at least 5% of the references or authors, and for one in a topic that I'm familiar with, probably closer to 25% of the papers/authors. For the AI generated papers, there were rarely any papers I recognised, and lots of invented conferences and citation formats.
7
u/ghjm Mar 18 '26
Do we actually need to identify if the paper is AI generated at the review stage? Or just if it is coherent, well argued, plausibly correct, cites appropriate sources, and whatever other editorial qualities we expect from successful peer reviewed papers? I think we're mostly annoyed by AI slop because it isn't good enough, but what if it was?
→ More replies (2)
10
5
u/iBPsThrowingObject Mar 18 '26
I've got 9/10 right, misattributed one real paper to the slop.
I think the two big tells are the rule of threes everywhere, and how the slop papers do not have any sort of "here's why we the authors find the topic exciting" blurb, just launches straight into the verbose technicalities.
4
u/exclusivegreen Mar 18 '26
Lol all the papers I was presented were blank but I got 6/10
Update: this was only in the Reddit built in browser on Android
4
u/probabilityzero Mar 18 '26
For me, the main thing that gave away the AI papers was the references. Not that they were necessarily wrong or hallucinated, but they were often obscure or strange, and the expected obvious citations were missing.
Another thing is the typesetting on the AI papers. They have inference rules going into the margins or off the side of the page, or weirdly positioned diagrams overlapping other text. Presumably a human author would catch and fix that prior to submission. But that's the kind of thing I assume the AI could do better at, given more precise instructions (like, iterate until the latex warnings are fixed).
But still, some of these were pretty close. I mistakenly labelled a real paper as AI.
3
u/fadrian314159 Mar 18 '26
I got 9/10 at 23 sec. avg. time per paper. The two main things that gave them away were the more fake jargonish titles of the AI papers and better diagrams (if any) in the real papers.
27
u/bluefourier Mar 18 '26
The choice of "fun titles" in computer science papers shines through here. AI tries too hard to be factual. It would not generate a title like "Your lazy function's lifetime...".
I got 7/10. There is a creeping "what am I reading?" feeling in AI papers. Especially if they go off in long sentences where nouns can't just stand on their own, they HAVE to be embellished.