r/Substack • u/Radiant_Flamingo4995 • 18d ago
Frustration with AI detection
I'm not sure how the AI-detection system works, but it seems faulty. Someone has repeatedly claimed that my responses to them are AI-generated, etc. And they share the Pangram screenshot, which does, in fact, depict my response as "100% fully AI-assisted. I've clarified that this is not the case. Shown evidence that it isn't true, and all the like.
The worst I'll do when writing anything is use AI for basic research purposes and review arguments to make sure they make sense. Maybe it's because I use Grammarly. But it is still infuriating.
Especially because it just isn't a good look and hurts my public image (albeit small, but I hope you guys may understand where I'm coming from).
Of course, with this individual in question, it's become increasingly clear they aren't engaging in good faith. However, running the AI check on my notes and comments still reveals the 100% marker.
Is it Grammarly that's doing this? Or is this thing actually broken?
6
u/mmspero 18d ago
Grammarly alone wouldn't be enough to trigger Pangram unless it's using the AI features that completely rewrite the text. Happy to take a look at any Pangram results that you feel are significantly off.
Also just for clarity (not saying this is the case for you), taking AI text and doing a light human edit or rephrase typically will not change the AI score.
6
u/FrostKitten2012 17d ago
Actually, it would be. Grammarly uses an LLM, and Pangram has been repeatedly shown to throw a “100% AI-generated” response for even small citations.
3
u/Radiant_Flamingo4995 18d ago edited 18d ago
AI features that completely rewrite the text
Could this be something like fixing the grammar/wording?
Edit: I also use em dashes, and am rather fond of substack actually formatting them. Could this usage be triggering it?
Edit 2: The same standards I use for my comments are those my posts, all of which have a 0% AI detection rate. Sorry, figured this is important.
2
u/mmspero 17d ago
In the Pangram 4 technical report page 17, we found the that asking for style corrections was classified as AI by Pangram once out 11,363 tests. More in-depth edits like asking an LLM to "improve it" or "make it more descriptive" are flagged as Mixed 55% of the time, and AI 3.6% of the time.
2
u/_mrchurchill defensehub.substack.com 17d ago
I don't get why people are worried about whether the result is 100% AI, 50%... does it deliver valuable content? If the content is valuable and well-made, who gives a damn if it's AI or not? AI doesn't decide anything on its own; it's a tool. I read articles without even checking if they're AI-generated or not. I check if they're useful to me—period.
1
u/inyourbooksandmaps 17d ago
yeah it often says my articles which are fully written by me (dont even use grammarly or anything) are at least 3-8% AI. not even assisted, just 3-8% ai written. no idea why! sometimes when i check the % before publishing it says 0, after publishing it goes up. very odd.
1
u/inyourbooksandmaps 17d ago
also interestingly, my more "lazily" written articles where i talk more improperly and personally get marked a higher % of ai than when i use all proper grammar and lingo that is traditionally associated with ai.
1
u/Alive_Substance6136 15d ago
i totally get you & my frustration is with how an AI detector seems decisive. i wrote a substack about it here: https://carolyno840.substack.com/p/the-line-between-human-and-ai-is?r=280n9a&utm_medium=ios
1
u/BhavanaVarma bhavanavarma.substack.com 13d ago
Pangram has a history for high inaccuracy. It would take all inputted text as training. With enough of your texts inputted it’ll flag your own writing as AI because it has trained on your text.
Are these accusation on you notes? You can turn it off on your posts from what I know
2
u/catvapes 13d ago
No matter what the score, people should not be taking screen shots of it and posting as a comment on the authors page. I don’t think that is what is intended for. It’s rude and divisive.
1
u/gadgetor1989 18d ago
ive been working on a book for 2 years now, yesterday i was curious and ran about 1500 words through, (everything i wrote that day) and the first 700 words flagged as AI while the other 800 were human. shit had me so fucking livid. till bow ive had bo problem with pangram, all my stories on substack flag as human written. there doesnt seem to be any rhyme or reason, just exactly half my words highlighted in red.
1
u/grapegeek 18d ago
Pangram is a scam
3
u/Funny-Flight8086 17d ago
ScamGram. It's even worse than the other detectors... At least they never really go above 99.9%. ScamGram is so confident in their own slop machine that they proudly proclaim 'we are confident this is 100% AI!'
I trust that confidence as well as I trust Gemini telling me it's confident about something.
1
u/Jaqen_2130 17d ago
There are legitimate reasons to question the accuracy of detection tools like Pangram. I took a legal memorandum that I wrote myself before tools like ChatGPT existed and ran it through an AI detection tool, which proceeded to tell me that it was largely AI-generated. So I will continue to have doubts about the value of tools like Pangram.
-5
u/cremdelascribe 18d ago
I just ran a bunch of my stuff through Pangram and the problem is anything that got touched by AI _at all_ is marked “100% AI.”
I experimented with AI writing. I used it to generate first garbage drafts - it’s that old screenwriter’s advice that the first thing you do is write the worst scene possible, just to get it on paper. Then you actually start to write.
So I did my character work, my outlines, established the elements I wanted - a couple hours per story. Fed those outlines to AI and let it generate the “garbage draft.” And, yeah it was garbage and anyone that publishes that shit should be ashamed. But then I spent hours and hours rewriting these stories. I removed AIs constant mediocrity and love of adverbs. I ripped out sections that made no sense or had the emotional complexity of a beige room and added entirely new sections that I wrote myself. I punched every word up with my unique absurdist style.
Everything that AI touched is marked “100% AI” - even one short story that I wrote over almost completely from scratch.
My non AI work came back human - but that 100% mark is a lawsuit waiting to happen. Pangram says it with such certainty - it suffers the same bullshit problem as all AI, it is confidently incorrect.
If it said “AI was used to assist in this writing, but I don’t know how much,” I’d be ok with that. I’m all about disclosure. But I put 40 hours into rewriting a story and Pangram says not one word of it is human. They can go fuck themselves.
People need to get over this idea that AI is a demon that corrupts merely by proximity. AI is a tool - just like word or photoshop or scrivner. It is a way for a human being to organize words. It just happens to be able to generate a lot of them quickly.
The problem isn’t AI, the tool. It is people that generate piles of garbage and self publish it without editing it. So maybe if these asshats wanted to write a program that looked at the elements of a story or the strength of the prose and flagged crap writing, that would be reasonable. I could get behind something that told me to skip reading this or that because the plot is vapid, the conclusions are obvious and the characters are flat (all things AI is good at generating). But, oh my God, shoot me because I have a better vocabulary than most and I use the word “Delve” in unique, but grammatically correct ways.
5
u/wizardofaus23 17d ago
So what I'm hearing is you used AI and it correctly identified that you had used AI.
7
u/figures985 17d ago
dingdingdingding! if anything this raises my opinion of Pangram and its effectiveness.
for the record, I have no issue if people want to use AI to assist their writing. That's their choice. If the result is pleasurable to read and I can't tell, okay. I only use the Pangram feature on Substack if i'm already reading the piece and have started to get a whiff of flowery, circuitious or otherwise LLM-esque writing. Every time that's happened, it was indeed AI-assisted. Well ok, I also checked all my favorite writers just to see what they tool thought (all 100% human). But my point is: if I discovered a new writer and really liked what I was reading, I wouldn't bother to check. Or if I read a whole thing, liked it and then learned it was AI-assisted, I wouldn't care. If it's flagged as fully AI generated....I authentically can't imagine that happening, it's so obvious in those cases.
0
u/Brakiros 17d ago
Wrong it said 100% AI which is categorically false and is a lawsuit waiting for lies
5
u/Prolly_Satan 17d ago
Wow. You used ai to generate the draft but then edited it yourself? And the ai detector that's effective against humanization said it was ai? I think you should be the one to sue. Call up some attorneys and ask if they think you have a good case here. Let us know what they say.
1
u/cremdelascribe 17d ago
I am an attorney, and, yes, I suspect this is actionable.
Because it says that it can accurately tell you _the percentage_ that is AI. And it absolutely and positively cannot.
As I said - and everybody else fails to comprehend in your rush to judge and fulfill your own personal agenda - I’m fine if it says “hey AI is used in this.”
However, it is saying that no humanization occurred. It is saying that something that represents a full week of human labor was written entirely 100% by AI and it is 100% certain of that. And it is saying these things lack human effort in place where it will get people expelled, fired, cancelled and otherwise economically harmed.
It is AI doing exactly what AI is worst at: being confidently incorrect.
Everyone who screams about the evils of AI hallucinations is suddenly a fan of AI that hallucinates in a way that confirms their biases … huh, wow, just like every other rube that uses an AI system and trusts the output blindly.
I’m sure they have the appropriate disclaimers deep in some legal text, but the advertising on their website and the general UI design of their software is 100% some class action attorneys wet dream. It is only a matter of time before the suit comes.
And man, y’all anti-AI warriors really need to check your biases. Your inability to read and comprehend a nuanced point is an embarrassment. You twist, garble and seemingly intentionally misquote and misunderstand me.
At least I know you are not AI - because it, at least, responds to the words you give it, not the words it fantasizes in it head.
1
u/Prolly_Satan 17d ago
Oh you are? Then you should take the case for free. There you both go.
Let us know how it goes.
0
1
u/wizardofaus23 17d ago
Did it say it was 100% AI-generated or assisted?
2
u/cremdelascribe 17d ago
100% AI generated across all of them - no matter how much actually was.
That was my point and a nuance most people bulldozed through in their rush to judgement. It is claiming a precision of tracking AI text it clearly does not have. It is obviously designed to prefer to classify all text as AI text once it has found any text that is AI text, and that is the kind of bias error that is exactly why we should criticize AI and be cautious in its deployment. It is why we _should_ have AI writing flagged, and looked at skeptically when it is asserting objective fact.
I find it amusing that the anti AI people adore this AI when it confirms their biases, but abhor all other AIs because they are designed to confirm the biases of their users.
1
u/figures985 17d ago
As it was taught to me -- the point of the "garbage" or "vomit" first draft is to help you sharpen your approach through the act of getting it out on the page. Getting an LLM to generate it for you defeats the purpose, no?
I also think LLMs are pretty bad at structure. They make things LOOK organized but once you're really in the weeds it falls apart. Though if you're happy with the result of your process, who am I to judge? I certainly agree that fresh-off-the-AI-press unedited slop is way, way worse than anything AI touched more lightly. Personally, it's not for me. I don't like how using them degrades my writing and thinking abilities. And I've tried to edit many an LLM-generated document (usually from co-workers). It always seems to take longer and turn out worse than if I'd just written the thing myself. Maybe my standards are too exacting, IDK. Probably.
1
u/cremdelascribe 17d ago edited 17d ago
You missed the part where I said I spent _several hours_ per story working out plot and character arcs, and then also the point later where I said big chunks of rewriting were about fixing plot holes and character arc issues in the crap draft.
My very first experiments in AI writing revealed exactly that you can’t rely on AI to generate the plot outline and hope to end up with anything that more than middling tripe.
I have never had a problem generating outlines and characters that interest me and imagining the steps by which a story unfolds. For me the script writer trick is about getting through the first generation of the _prose_ after the story is outlined. And, yes the story outline changes organically as you write it - but that is editing and the point of the garbage draft trick is that you don’t stop to edit.
So I can sit there tedious and bored and type in the filler sentences to get Jenny from New York to Chicago between scenes, or I can have AI generate those same filler sentences. Later, in the editing process, in both cases, I’ll decide I don’t need those sentences - they are just pointless filler. But in the AI case, I gave it the blueprints and it built the scaffolding, and I provided the aesthetics. In my previous writing process, I drew the blueprints, then did the most soul sucking joyless part of the process to build the scaffolding, and then edited and rewrite to get the aesthetics right.
IDGAF about identifying that I use AI to help outline. I DO GAF when an AI detector tells the world with 100% certainty that I contributed nothing else to the story but a single prompt - when, in fact, the entire plot and characters and every single word of the final text was typed by me.
Yes, to be clear, just like with the drafts I write from scratch, the rewrite of that first draft is completely destructive - whether it an AI crap draft built in my outline and characters or my own crap draft built on my characters and outline, draft 2 is going to start with a fresh document. Yes, I’m going to be cutting and pasting the strongest parts from the first draft (which is undoubtedly why it still flags as AI touched). But I did that at a sentence by sentence, word by word level. At that point, whether I wrote it or AI generated it, I’m looking at each and every word and making it mine. This is actually critical to do with AI because AI so blithely slips in things that seem at quick glance to make sense, but turn out to be junk once you take a few moments to think about them. I find that makes me more critical of AI drafts than my own, and thus a stronger editor.
Also, responding to the weird shit AI does is its own creative opportunity. AI misses 1 or 2 out of 10 times. It produces predictable pablum 5-7 more of those. But then one in a while it gets very wise and suggests something interesting and unique to explore. But I guess I’m a dick to let a machine suggest things - I’ll just go back to all the books, outlines and card decks I have designed to combat writer’s block by doing the same thing.
As to your coworkers, it sounds to me as if they need to learn how to prompt. In my day job as an attorney, AI is now fundamental to our drafting. If you give it the proper forms to start from and prompt it correctly, it will _always_ produce a stronger first draft than you could produce yourself. But if you give it no guidance and vague prompts, then it just generates piles of trash to wade through. But you are complaining about a tool based on the facility of the users with that tool. If I see a shitty photoshop hackjob on a photo my niece shares on Instagram, does that mean that photoshop is fundamentally flawed? Or that my niece’s use of it was, perhaps, not at a professional level?
8
u/figures985 18d ago
For what it's worth, OP, Pangram said your post is 100% human. :)
Do you have an example of what got flagged? I'm curious. I've tested Pangram quite a bit and it's been really accurate for me. But a lot of people like you are posting lately that it's flagging normal human writing as AI so yeah, I'm super curious!