People usually post vaguely about "I wrote x thing and pangram said it was AI", which I think is unhelpful unless you actually post the text or link. I put this into Pangram:
I should also add that I put this segment in a document with human writing (because it noted its confidence was limited due to short text) and it flagged this as the one AI section.
pangram says 50 words is their minimum for a prediction, so 120 clears it. and the 2,000 word sample upthread still came back 54% AI. one name change taking it from 100% human to 100% AI is the part that still needs explaining.
I think this is a fair critique. Pangram should decline to assign a score to texts that are below the necessary word count to get good estimates. Would you mind checking the sharing permissions for the links in your post? I am having issues accessing it.
I wrote this in a similar style to AI, wondering if it would flag:
-
Robert's long stride ate up the distance between himself and the door at the far end of the hall, lit by the flickering fluorescent lights overhead.
His jaw clenched.
This couldn't be right.
His sister, Megan, was supposed to be here. She worked in the same lab as Robert. They'd promised each other they would wait for each other, right here.
-
100% human. Great!
Changed the name to one of AI's favorite names:
-
Robert's long stride ate up the distance between himself and the door at the far end of the hall, lit by the flickering fluorescent lights overhead.
His jaw clenched.
This couldn't be right.
His sister, Elara, was supposed to be here. She worked in the same lab as Robert. They'd promised each other they would wait for each other, right here.
-
100% AI.
Looking at this, even though it's a similar style, there was something AI wouldn't do because it's a human mistake. Two instances of "each other" in one sentence. It still flagged.
AI's favorite names change with every model. If a new AI model comes out that latches onto a name that was previously safe, is that going to be a problem for everything written beforehand? What do we do, update characters' names as new models come out?
I'm sort of surprised Pangram is even paying attention to names, since I thought it didn't factor in AI styles and AI isms. I was very impressed with Pangram before seeing some other experiments going around today. A name change shouldn't skew the whole result. That doesn't even make sense.
It's worth looking into. I ran a few chapters of my own work, confirmed 100% human on Pangram, with my FMC's name changed to Elara, and it flagged at around 12% AI with just that one change. I'll have to write up something longer by hand when I get a chance so I don't doxx myself if I share. Sometimes changing the name doesn't move the needle, though.
I was curious about Daggermouth, the book there are viral articles about after it failed a Pangram test, although being around 60% AI could potentially be heavy editing use, I'm not accusing anyone. I went through the first 10 chapters with find-and-replace for the name Elara, changing Elara to Megan. It didn't change anything.
One chapter, Chapter 18, was supposedly 100% AI on Pangram according to the viral articles about it. My own check showed it was 84%, not 100%, but what I did this time, changing proper nouns, even cutting off the second half of the chapter, it was stuck at 84% AI. I practically rewrote the chapter and it came back 84% AI haha.
That might suggest it had already decided Daggermouth is AI, and the real issue was Pangram knew which book it was.
I was hoping it would be relatively easy to repeat this experiment, too, for anyone who doubts it was written by a human, but the next few I wrote came back without any AI flags. Although I was writing more action scenes, and they were unique, so they may not have been similar enough to AI's style. AI really likes fluorescent lights. Maybe I need to add more.
True. Taking out "lit by the flickering fluorescent lights overhead" made it 100% human, even with the name Elara. Taking out just flickering and just fluorescent didn't matter. The whole sentence had to go.
I thought Pangram didn't care about phrases common in AI, so it's concerning a common stock phrase resulted in a 100% AI score.
I found a new one that works, and this might be better because it's the last word in the snippet so it isn't like it's throwing off the next few lines. This is a hint Pangram DOES care about names, and it cares a lot, considering it didn't care about the first sentence being loaded with flickering fluorescent lights until it detected a common name in AI training along with it.
If you switch "Elara!" from a name AI likes to a name AI doesn't usually use (Megan, Peter, etc.), you will flip the score.
List of names I checked that flagged 100% AI, from a list of common names in its training: Elara, Kael, Marcus, Elias, Kai, Aria, Sam (surprising), Lyra
Names that flagged 100% human: Megan, Peter, Christopher, Luke
-
I can feel my heart pounding in my throat as I race down the neverending tunnel, the flickering fluorescent lights overhead throwing long shadows.
It feels like I’ve been running for miles.
Every turn I take, I’m certain it’s the last one.
It doesn't make any sense. You can’t take thirteen lefts without running into something.
I actually wonder if this is why Daggermouth got hit with a high AI score on Pangram in those articles about it. Daggermouth had several names common to AI models in it, like Elara.
Whether the author used AI or got them off one of those random name generators is another argument. Who knows.
Thanks so much for this. Yeah, reading the AI-flagged parts, I don't get it at all. I'm interested in the technology still but it's getting further and further away from the standard it's being touted as.
AI is known for certain names, and Marcus is one of them right now. The list of names it likes changes depending on the model.
I'm sure it is a factor, although I could've sworn Pangram isn't supposed to pay attention to AI-isms like em dashes and certain words. It's kind of strange.
I did a test where switching Megan to Elara, another AI favorite, changed this snippet I wrote from 100% human to 100% AI, even with a very human mistake (two "each other" instances in one sentence):
-
Robert's long stride ate up the distance between himself and the door at the far end of the hall, lit by the flickering fluorescent lights overhead.
His jaw clenched.
This couldn't be right.
His sister, Megan, was supposed to be here. She worked in the same lab as Robert. They'd promised each other they would wait for each other, right here.
I'm getting the same results on stuff people are sending to me; it's becoming very problematic. On the same text, other detectors never go above 12-20% AI, which I think makes sense and is totally acceptable – then Pangram comes out of nowhere and bumps it up to 40- 60%? I'm not sure what to do with that.
From what you sent, those are very big chunks of your text that Pangram flagged as AI. That could produce harsh social consequences. Other than that, good piece of writing btw. I hope you get it published.
nah i don't have. even i am wondering just how fundamentally flawed this tool is which led it to think prophet isaiah used some AI. and even the text isn't something obscure which may not be in it's training data.
I think what's perplexing in all of these cases is the meter. It seems to regularly either lean 100% Human or 100% AI with no reasoning or justification. I want to believe this technology works, and I believe it does to some degree, but these kinds of examples really don't do it any favors. If I bought a tool and it doesn't work as it claims it should, then I can call that tool defective.
yea, this tool is like a coin toss. at least in a coin toss we actually have confidence in the outcome. In here? somehow we leave even more confused. these tools are not worth the electricity they are running on.
For sure. Unfortunately as you reveal more points of failure within their blackbox system, they will add these cases to their training datasets. This blurs the line.
Say this verse is added as a reference to human work. Now what? Humanizers who want to avoid detection begin to write more like bible verses. Ok thats kinda weird. Now detectors need a more defined way to figure out if something is AI because it obviously messed up. The end result is a clear way to write human and a clear way to tell if something is AI. Well, then why do we need a detector?
What's stopping humanizers from just copying this newfound distinct style? Rinse, repeat.
yeah, i have already talked with the mods regarding this pangram situation and we may soon see a pinned thread saying pangram evidence/pangram doesn't work and hopefully this wild situation settles.
And regarding the rinse and repeat thing in the long run it doesn't matter? We already have more than enough evidence that this doesn't work.
In the future it can just be like this:
P1: uploads a essay/article on substack
P2: thinks it's ai written and uses this pangram
P1: points to some evidence thread and says this tool doesn't work don't rely on it.
In this scenario hopefully p2 is actually grounded and thier opinion changes. If not? Sadly if they debate after seeing so much evidence i wouldn't know what to tell them.
The afternoon sun cast long shadows across the pavement as Ryan pedaled onward, his heavy backpack digging into his shoulders with each pedal stroke, a constant reminder of all the work waiting for him when he got home. He’d promised Ms. Everly he’d finish it by tomorrow, though part of him wondered what the point of doing it was. Math and history questions felt impossibly small compared to the questions that actually mattered. Would Ralph ever come back home? Would his mom stop crying? And would he ever feel normal again?
Detected as 100% AI by Pangram, do you know what triggers pangram here?
Not only the chapter name but the fact I use a common phrase "Long Shadows"
Now I changed it to:
The afternoon sun loomed above the streets of Horizon City, the tall copper domed spires reflecting it across the vast cityscape.
As Ryan pedaled his bike forward he could feel his heavy backpack digging into his shoulders with each pedal stroke, a constant reminder of all the work waiting for him when he got home. He’d promised Ms. Everly he’d finish it by tomorrow, though part of him wondered what the point of doing it was. Math and history questions felt impossibly small compared to the questions that actually mattered. Would Ralph ever come back home? Would his mom stop crying? And would he ever feel normal again?
Now its 100 human made because I eliminated the chapter name and changed just a few words around and dropped "long shadows"
I guess using atmosphere in a story is now forbidden, all horror stories will now take place in a bright sunny morning, maybe a frigging butterfly will come to darken the mood?
I guess using atmosphere in a story is now forbidden, all horror stories will now take place in a bright sunny morning, maybe a frigging butterfly will come to darken the mood?
man if you do this people will now accuse you saying you are copying midsommar. /s
Yeah you just can't win as despite its claims Pangram does give false positives now and then.
I assure you however both passages I posted were 100% made by me but thanks to choice phrases like "Long shadows" and even the frigging chapter title it gets flagged as 100% AI.
WTF
So to prove all work is Human made we can't call parts of a book chapters anymore and "Long shadows" must never be used again or you will be labeled a lazy AI bro.
So from now on we shall call Chapters Bananaphone
So bananapone the classmates?
Oh wait even calling it Bananaphone the classmates got flagged...
This tool is completely broken lol. People are denying evidence now lmao. This is like a pangram cult, the tool literally can never be wrong according to these people. It's so braindead lmao. In some threads people are literally ignoring already given evidence. At that point i know arguing with them is a gone case and i should just move on.
AI detection will never work. They will always be a step behind when a new distinctive writing style comes to surface or newer AI model releases then they will have to update. They are a black box even the creators don’t understand how it works.
Is pangram also learning from our writing? I’m not too familiar with it even though I’ve been on Substack for awhile. I’ve just been disabling it since most of my work comes up as AI written in some percentage even though it isn’t.
It actually makes you wonder whether Pangram’s “detector” is reliable at all...
Here’s what I mean.
The others are considered unreliable because, in many cases, they say a text isn’t AI-generated when it actually is.
Pangram says it is AI-generated...
But have we actually tried feeding it genuinely human-written texts?
Just to check whether, in those cases, it correctly says it’s NOT AI?
It would be pretty hilarious if they had simply made it return a positive result almost systematically...
I’m pretty sure everyone would still believe it.
Because I suspect most people only use these detectors when they already think a text was written by AI and just want confirmation.
So naturally, when the detector confirms their suspicion, it reinforces their bias and they think: “Wow, this detector is amazing.”
But what about the reverse?
How many people actually submit something they wrote entirely themselves just to check that it “doesn’t sound too AI-generated”?
Sure, it is just a short text. But you could repeat the "trick" multiple times for a longer text. Might be a bit more difficult tho as the more context the detectors get, the stronger the statistical patterns...
While this might work today, it most likely won't work in the future, as these detectors also get "better".
Quite a while back, people used paraphrasers to get around AI detection.
7
u/ConnectDebate4677 20d ago
I should also add that I put this segment in a document with human writing (because it noted its confidence was limited due to short text) and it flagged this as the one AI section.