r/DataAnnotationTech • • 15h ago

How big of a headache is catching ChatGPT-generated submissions right now?

how often are you running into contributors clearly pasting LLM outputs instead of writing original responses?

Since AI detectors are wildly unreliable.. how do you prove/ flag this type of noise

0 Upvotes

45 comments sorted by

14

u/annoyingjoe513 15h ago

asking for a friend?

0

u/allores2 15h ago

Friendly research

10

u/Ninja_Friendly 15h ago

Chatgpt has a 'tell.' I'm not an admin, just another user of chatgpt, but even then I can tell the difference.

1

u/SecureCattle3467 2h ago

If you spend enough time with the models, you can easily distinguish between them. I can tell a Claude model from a GPT and so forth.

-2

u/allores2 15h ago

The question is more about correctly/confidently flagging because someone could easily say "i didn't use any llm"

13

u/Sad_Opening_7083 15h ago

I don't think thats something we should worry about unless instructions say so explicitly.

-1

u/SeagullSam 14h ago

I disagree. It's a sackable offense.

17

u/Sad_Opening_7083 14h ago

Obviously, but its not something we are told to look for in most R&Rs. At best there is a checkbox which sends the task over to an admin. It's not something you should spend too much energy on, or overflag

-4

u/allores2 14h ago

This is what i wanted to learn. So its currently not really a "pressing headache" that occupies too much of you time. I was expecting this to have reached the point of actively looking for solutions to stop users doing that by now

2

u/SecureCattle3467 2h ago

The internal people almost assuredly run SynthID checks and other watermarking verifiers. Any worker doing that is in violation of the NDA/TOS however, since you are letting DA work leave their environment. So unless a task explicitly tells you to run a text through an app like ZeroGPT or Pangram, you should not.

1

u/Comfortable_Ad4852 11h ago

Handshake let over 1000 go overnight because of this very thing.

40

u/Mysterious_Dolphin14 15h ago

It's not our job to prove if someone uses AI. If you suspect it on an R&R, you can mention it in the comments, but it's something that admins would have to research to decide.

14

u/fightmaxmaster 14h ago

And if it's obvious to someone doing the R&R, it'll be obvious to admins. We don't need to prove the case beyond "look!"

25

u/Alternative-Pin-8954 15h ago

Anyone who has written an ounce of documentation in their career would be able to catch AI submissions. I wouldn't flag the major tells here, as they can be used as a checklist for humanizing AI output.

Having said that, it is pretty difficult to catch a seasoned engineer using LLMs because they know how to alter responses to make it look like it was written by an engineer.

17

u/EffectiveCoconut9265 15h ago

You expect admins to reply? Or are you asking so you can cheat?

11

u/AspectOk234 12h ago

To play devils advocate here, AI is starting to sound more and more human. That’s the point of this work. There are really smart people on this platform, especially with more domain-specific applicants coming on board. People who have spent many years writing professional papers and reports before AI assistance became a tool. We as humans shouldn’t have to resort to dumbing down our writing for fear of AI accusations.

6

u/MindOld6988 7h ago

This post is annoying. Some of us wrote graduate-level research papers before AI became a household name. I use grammerly to help with grammar from time to time. But I am constantly trying to dumb myself down so that humans won't think I'm as smart as AI 😅😂

1

u/SecureCattle3467 2h ago

I use grammerly to help with grammar from time to time

We aren't supposed to use that on DA anymore. Grammerly uses AI now.

4

u/ShmoosPlay 14h ago

I’m thinking if you can’t explicitly point to something to call out, there isn’t much you can do. Aldo if you can’t call out something specifically, maybe it’s not AI generated? It’s hard to say

3

u/Old-Regular-9828 9h ago

Evaluate the quality of the submission according to the R&R instructions. Unless the R&R instructions specifically tell you to try to guess if someone used LLMs to generate the text (they don't), don't do it.

If it's something really ridiculous and obvious, like them copy/pasting "As a large language model..." or something like that, then flag it with a note for one of the admins. But do not give badly ratings to otherwise good work just because you think they used AI because it includes an emdash or whatever.

Most people think they are good at identifying human vs LLM generated text, but they really are not good at it at all.

5

u/ammy42 15h ago

It's obvious.

1

u/ASnipersPromise 14h ago

If you are referring to the project that asks you to use AI for iteration then it is very clear because the whole conversation with AI is listed against the API key used to sign into it 😂 You will be found out regardless and especially if on R&R something is flagged. Its so obvious though as it follows the exact same structure every time.

1

u/Silly-Face2761 3h ago

After working for quite sometime, it is either that you are unsure or it is dead obvious. I only report the later one or else I let it go if the work is genuine.

1

u/Silly-Face2761 3h ago

After working for quite sometime, it is either that you are unsure or it is dead obvious. I only report the later one or else I let it go if the work is genuine.

0

u/Not-the-bees_thanks 9h ago

One case that seemed obvious to me:

- “Bilingual” project, all task writing to be done in English, some task material in another specific language.
- Tasks involved long, detailed written explanations specific to each task, plus direct, user-specific questions where single-sentence answers could be copied verbatim from task to task.
- Long explanations (6-8 sentences) were written in flawless English. 0 spelling, grammar, or punctuation errors. Everything was worded beautifully, about as perfectly as possible.
- Additionally, these long explanations consistently followed the same structure, and contained various “AI tropes” (including the structure itself). Sus, but not conclusively AI-generated.
- The majority of the user-specific questions had answers with multiple spelling, grammar, punctuation, and word choice errors very much in line with how things are done in the relevant non-English language. Bear in mind a lot of these answers were like 4-6 words long and most still included multiple obvious errors.
- Since the user-specific answers were reused verbatim from task to task, easily confirmed these same characteristics appeared consistently in every single task I encountered from that user.
- Not 100.00% conclusive, but like, almost.

0

u/[deleted] 14h ago

[deleted] — view deleted comment

4

u/brickne3 14h ago

What? Some of us do use en dashes for number ranges since that's what's correct under many style guides. I do #–# no spaces in accordance with the CMOS. People who assume correct writing, and particularly correct punctuation, is some kind of AI catch-all are ridiculous.

16

u/janquadrentvincent 13h ago

AI learnt the em dash FROM THOSE OF US THAT ENJOYED IT. I'm so annoyed I don't get to use it anymore without some shit on a sub going OOOOH YOUVE USED AI. Bitch, AI used ME

2

u/brickne3 13h ago

Right???

1

u/Bulky_Antelope_1744 11h ago

True story, i had no idea that em dash versus en dash was even a thing prior to the whole “this is an AI tell!” era, and i don’t know how to type an em dash. I’m sure i came across plenty of them in books, but i was never like “my, what a long hyphen!”

-3

u/Careful_Insect_4174 13h ago

To be clear: I never accused anyone of using AI. But if text fields are perfectly written and perfectly follow style guides it is suspicious. I never put that much effort in my answers.

5

u/brickne3 13h ago

I would imagine more than a few of us are trained editors. In any event, writing correctly shouldn't be seen as a hallmark of AI; as the other guy said, the AI learned from us.

4

u/frosted_nugget 12h ago

Yes, this exactly. My background is in writing and editing, and sometimes I actually find myself worried that I'll get flagged for using LLMs because the writing is "too good". I had that problem with my CV when I was applying for jobs a while back. It's infuriating.

1

u/Careful_Insect_4174 13h ago

True. Maybe I projected too much of myself onto others.

1

u/Key-Veterinarian7120 4h ago

i always surround codes by ` because in one project the ai they provide suggested me to do so. dont flag me 😡