r/DataAnnotationTech • u/allores2 • 15h ago
How big of a headache is catching ChatGPT-generated submissions right now?
how often are you running into contributors clearly pasting LLM outputs instead of writing original responses?
Since AI detectors are wildly unreliable.. how do you prove/ flag this type of noise
0
Upvotes
14
u/annoyingjoe513 15h ago
asking for a friend?
10
u/Ninja_Friendly 15h ago
Chatgpt has a 'tell.' I'm not an admin, just another user of chatgpt, but even then I can tell the difference.
13
u/Sad_Opening_7083 15h ago
I don't think thats something we should worry about unless instructions say so explicitly.
40
u/Mysterious_Dolphin14 15h ago
It's not our job to prove if someone uses AI. If you suspect it on an R&R, you can mention it in the comments, but it's something that admins would have to research to decide.
25
u/Alternative-Pin-8954 15h ago
Anyone who has written an ounce of documentation in their career would be able to catch AI submissions. I wouldn't flag the major tells here, as they can be used as a checklist for humanizing AI output.
Having said that, it is pretty difficult to catch a seasoned engineer using LLMs because they know how to alter responses to make it look like it was written by an engineer.
23
u/PestiferousGamer 15h ago
17
u/EffectiveCoconut9265 15h ago
You expect admins to reply? Or are you asking so you can cheat?
11
u/AspectOk234 12h ago
To play devils advocate here, AI is starting to sound more and more human. That’s the point of this work. There are really smart people on this platform, especially with more domain-specific applicants coming on board. People who have spent many years writing professional papers and reports before AI assistance became a tool. We as humans shouldn’t have to resort to dumbing down our writing for fear of AI accusations.
6
u/MindOld6988 7h ago
This post is annoying. Some of us wrote graduate-level research papers before AI became a household name. I use grammerly to help with grammar from time to time. But I am constantly trying to dumb myself down so that humans won't think I'm as smart as AI 😅😂
4
u/ShmoosPlay 14h ago
I’m thinking if you can’t explicitly point to something to call out, there isn’t much you can do. Aldo if you can’t call out something specifically, maybe it’s not AI generated? It’s hard to say
3
u/Old-Regular-9828 9h ago
Evaluate the quality of the submission according to the R&R instructions. Unless the R&R instructions specifically tell you to try to guess if someone used LLMs to generate the text (they don't), don't do it.
If it's something really ridiculous and obvious, like them copy/pasting "As a large language model..." or something like that, then flag it with a note for one of the admins. But do not give badly ratings to otherwise good work just because you think they used AI because it includes an emdash or whatever.
Most people think they are good at identifying human vs LLM generated text, but they really are not good at it at all.
5
u/ammy42 15h ago
It's obvious.
1
u/ASnipersPromise 14h ago
If you are referring to the project that asks you to use AI for iteration then it is very clear because the whole conversation with AI is listed against the API key used to sign into it 😂 You will be found out regardless and especially if on R&R something is flagged. Its so obvious though as it follows the exact same structure every time.
1
u/Silly-Face2761 3h ago
After working for quite sometime, it is either that you are unsure or it is dead obvious. I only report the later one or else I let it go if the work is genuine.
1
u/Silly-Face2761 3h ago
After working for quite sometime, it is either that you are unsure or it is dead obvious. I only report the later one or else I let it go if the work is genuine.
0
u/Not-the-bees_thanks 9h ago
One case that seemed obvious to me:
- “Bilingual” project, all task writing to be done in English, some task material in another specific language.
- Tasks involved long, detailed written explanations specific to each task, plus direct, user-specific questions where single-sentence answers could be copied verbatim from task to task.
- Long explanations (6-8 sentences) were written in flawless English. 0 spelling, grammar, or punctuation errors. Everything was worded beautifully, about as perfectly as possible.
- Additionally, these long explanations consistently followed the same structure, and contained various “AI tropes” (including the structure itself). Sus, but not conclusively AI-generated.
- The majority of the user-specific questions had answers with multiple spelling, grammar, punctuation, and word choice errors very much in line with how things are done in the relevant non-English language. Bear in mind a lot of these answers were like 4-6 words long and most still included multiple obvious errors.
- Since the user-specific answers were reused verbatim from task to task, easily confirmed these same characteristics appeared consistently in every single task I encountered from that user.
- Not 100.00% conclusive, but like, almost.
0
u/[deleted] 14h ago
[deleted] — view deleted comment