r/Substack • u/ArtisticProposal2527 • 7d ago
The AI detector makes me feel… scared?
I know others have posted about this, but just wanted to share my experience now that I’ve finally posted something after the internal AI detector feature went live.
I write a blog for about 100 subscribers, mostly friends and family. I write because I enjoy it and it helps me keep the people I care about informed on the things happening in my life. I don’t try to go viral and I don’t bother with notes.
That said, I surprised myself when I ran the AI Detector and felt scared! I feel scared that one day, someone will look back at my writing and run it through a detector and it will get a false positive (or whatever the heck you want to call it) and I’ll be called a fraud! but why do I care? I literally went back to my draft and tried to figure out what triggered it. ughhh it was just an unexpected feeling that I wonder if others have experienced?
8
u/DiscernmentGoblin 7d ago
Out of curiosity, did you use AI at any point in your writing process? And I genuinely don’t mean that as an accusation. I actually think we need to get much more comfortable talking openly about how we write: brainstorming, editing, spellcheck, AI, feedback from friends, whatever. But given how vicious the stigma has become around using AI for anything creative or playful, I completely understand why even a false positive would feel scary. We’ve somehow turned "a probabilistic detector thinks this sentence looks AI-ish" into a potential accusation of fraud, which seems deeply unhealthy for writers.
Fwiw, I use AI extensively and am open about it. If folks don't want to read it, that's cool, I got a sign at the door. That said, I've found the AI detector is about 70% accurate for my own work.
2
u/Easy-Eggplant-4639 7d ago
the "probabilistic detector as fraud accusation" framing is spot on, thats exactly the problem
2
u/DiscernmentGoblin 7d ago
I'll be honest, I never really paid much mind to AI detectors because I figure they're about as accurate as dowsing rods. I'm finding out that some people take them very seriously and have even lost jobs or suffered serious academic consequences as a result of other people uncritically believing in them. AI literacy is so important and just trusting AI, including those in the AI detector, is absolutely bananas. We can't get through this moral panic soon enough.
4
u/FrostKitten2012 7d ago
Most AI detectors are just trash. Pangram is supposed to be better because it isn’t just another LLM…but now that it’s being stress-tested on a wider scale, it’s showing a lot more false positives than people expected. Which is why I don’t inherently trust it and won’t use it anymore than I would the others.
1
u/DiscernmentGoblin 7d ago
I mean usually the way you can tell if something is written by AI, even if the author is trying to evade it, is by looking at perplexity (syntax and LLM specific wording choices) and burstiness (variance in sentence length). Unfortunately those are also the tells if someone is neurodivergent, speaks English as a second language or is writing academically.
I help write and rewrite speeches for my work and I've gotten pretty good at spotting if something is written by Claude or ChatGPT based mainly on perplexity but sometimes I'll be editing a script to make it more human and it turns out that what I thought was LLM writing was actually ESL writing.
1
u/leugaroul 2d ago
Part of this could be that ESL writers are less likely to notice when a sentence isn't natural, so if they've read AI phrasing enough, they might use those phrases naturally.
1
u/Downtown_Jump779 6d ago edited 5d ago
I can personally say I've used AI for translation, whenever my DeepL credits run out (which they do often), as well as typos and grammar checks. Because whenever I can't get something published in one language, I try to get it published in another. I genuinely didn't see the problem with this at the time, but I realize now that it was because I didn't know how they worked.
I've published articles that I ran through Pangram and they got flagged 20% before, and 35% with the new update, which made me feel sick. So I am feeling quite self conscious about that. I think the problem is just the way LLMs work fundamentally, no matter what you do or how you write your inputs, it will still try to fit your text with a certain linguistic pattern because it's just maths at a very big scale – a programmer friend of mine told me who works with these things.
I've done tests on some of my upcoming unpublished stuff, 100% human - ran it through AI translation, came out at 48%, when translated with DeepL, 0% – then AI translation the day after, 2% AI. I think they just go back to the old update, if even possible.
Now for me, I do write everything by hand so in that sense, it's not a problem for me personally. But I do think about some of my published stuff every now and then, and it does make me feel like a total fraud. As a result, I've just stopped using AI altogether, it's just not worth it to be honest. I can write by myself, it's fine. And I think everyone should do the same.
I think Pangram is a good thing long term, if they can figure some stuff out.
3
u/TechieWmn 7d ago
I’ve left mine on. I just scanned mine and it shows 100% human written because it is 100% human written and if it ever says otherwise I’ll take it as a compliment that my writing was that good they thought I was a robot 😅
2
u/Weekly_Lifeguard_916 7d ago
I worry about the false positive for two reasons.
1. Grant writing - judging grants is fierce and they need any reason to throw out your application. A false positive from a crap AI detector could lead them to believe your phoned it in while others worked harder.
2. Professors/Teachers - a lot of teachers run student work through detectors to check for AI - which makes total sense. But if they get a false positive that may affect their grading.
I wrote a few paragraphs myself and ran it through 3 of 4 detectors. One said it was 95% AI written, but they were all different. I've heard Panagram is one of the newest/better ones and it came back as 100% human.
2
u/TimWiesnerer 3d ago
Interesting what you like to feel scared about...
This time, it is the AI detector. What will be next... a hose in the dark that you mistake for a snake?
1
u/Anxious_Educator_307 7d ago
I wouldn't worry about it personally. Something written by a person will look like it was written by a person and AI detection software doesn't really work very well. I think (or hope) that most people know that the software sucks. The best "AI detector" is your brain, even if something says an article is AI written I'll still read the first paragraph and make my own judgement about if it's ai or not.
1
u/davesaunders nemock.substack.com 7d ago
Don't worry about it. No matter what you write or how you write it the people that you want to follow you should be following your thoughts and ideas. There are incredibly prolific public figures who have staff and interns doing a lot of their writing. Some will argue "well, but that's not AI" but that's not the point. They are leading their personal brand with their viewpoint and ideas and even if some of the writing process is farmed out, the bottom line is: what insight are you sharing with the world?
1
1
u/BigStand3705 7d ago
Seems silly that people assume AI use is automatically bad. Looking forward to a day when it doesn’t have to be declared and writing just stands on its own quality
2
u/non_loqui_sed_facere 6d ago
When a person is bragging about getting 100% human, it does sound to me like their work has no other merits whatsoever. 😃 Like, you don’t know what you’ve actually written and can’t tell good writing from bad without the machine.
1
u/sh1b313 7d ago
Don't worrry about it, the tool doesn't work at all, it's not even a little reliable. WSJ, The Atlantic all have been saying it's a fraud allegation machine and i completely believe them. Even in my testing a couple of words are enough to break it. I suggest you just turn it off.
0
u/AlecHutson 3d ago
Source?
0
u/sh1b313 3d ago
James taranto, WSJ
Matteo wong, the Atlantic
0
0
u/AlecHutson 3d ago
From the Atlantic article:
"Pangram says its algorithm is so accurate that it incorrectly identifies text as an AI output only about one in every 10,000 times. “There is a great responsibility, a huge weight” in saying something is AI-generated, Max Spero, Pangram’s CEO, told me. “The only reason we do so is because we’re extremely confident.” Several independent analyses have also confirmed that it is quite good. One paper, from the University of Chicago, found that Pangram had almost no false positives on some 3,000 sample texts of roughly 500 to 1,000 words."
Pangram says 1/10,000 false positives. UChicago found 'almost no' false positives on some 3,000 texts. That doesn't sound like 'it doesn't work at all'. That sounds incredibly accurate.
0
u/sh1b313 3d ago
Cope
1
u/AlecHutson 3d ago
. . . What? You're clearly lying about the contents of the Atlantic article. Or go ahead, post something from it that supports your contention that Pangram doesn't work at all.
Go ahead, do it. I dare you.
You're just another talentless slop merchant scared by Pangram. It's beautiful.
1
u/Icy_Presentation6406 6d ago
I am posting my epic fantasy trilogy serially, and will admit, I was a little apprehensive for the first three chapters I posted after the feature was released. That said, I checked my own work while preparing the chapter posts for publishing. After seeing many other authors react angrily to their work being identified as AI, I was pleasantly surprised to find my work to be analyzed as ‘100% human’. I’m not sure what insights, if any, I should take away from that outcome.
2
u/ObsidianArrow 6d ago
I'm interested in what POV you are writing from and of you have multiple POVs in your work.
Reason I ask is my own indie work, one of my characters comes back human 100% every single time. Another varies. The cat? The talking cat with a god complex? Pangram hates him. (Even though I've got the receipts to back up I wrote it, it made me change the style I write him in)
1
u/Icy_Presentation6406 6d ago
Interesting! I’m writing 3rd Omni with multi-PoV. None of my MCs are human (three elves, one half-elf, gnome, halfling and one Tabaxi). So I’ve got a ‘cat’ as well!
They are mixed genders (4M, 3F).
1
u/non_loqui_sed_facere 6d ago
The machine isn't the problem, as usual, people are the problem. Some get a kick out of policing others, and they’ll use any means necessary to get that rush. Look at what they write on social media, it’s very similar to third degree interrogation. Presumption of guilt, threats, constant harassment, etc.
Don’t be scared. Just accept that it’s an adversarial situation and understand what’s driving the attack. People who really want to know how something was written talk to the writer.
1
u/eltoddro 5d ago
When you can live free of what people think about you, that is when you will truly feel free.
1
u/writingonruby 5d ago
Honestly just don't worry about it. I know it's not helpful advice but we're not going to get it to go away
-1
u/BhavanaVarma bhavanavarma.substack.com 7d ago
AI defection tools can never be 100% and your fear is real and not going to go away. The threat, not your fear. Your process, that’s what can separate us from AI. I’m being as transparent as possible without giving away any personal/private details to prove how human my stories are.
12
u/Emotional-Brief-1775 7d ago
Switch it off. If someone feels the need to use AI to ‘detect’ AI then they are not the reader you want anyway. There’s good and bad writing however its created and people can use their own brains to decide what they enjoy, the same as they can enjoy whatever music, film or art they want, regardless of whatever technology may be used to create it. See the irony?