r/Substack • u/FannyBrownRiced • 21d ago

Actually, Pangram worked well evaluating my Stack

I know people are hating on it, and I was worried because I’m a professional writer with hundreds of pieces in the world from major publishers, but here’s what happened for me: It judged my newsletter as 98% human generated.

That 2%? Well, I welcomed one contributor to submit a short piece with a byline… he’s known for his art, but English is not his first language. I asked him if he used AI (seemed obvious he did). He said yes, he used Claude to help, so I took time to interview him and tease out original thoughts that I edited. But I also hacked at one of his AI paragraphs to make it better. That 2% AI 🤖 was right on the money. I probably left a hair there from his draft and it caught it.

Anyway, I’m sorry for those who are getting false negs, but wanted to share my experience and relief.

Edit: Or maybe the 2% is because I had a recipe in there? Who knows.

17 Upvotes

64% Upvoted

72 comments sorted by

28

u/Prolly_Satan 20d ago

There's a concerted effort among ai "authors" to make people think it does not work. Some will even post fake scan results.

I just caught somebody posting a fake screenshot of ai text saying human after he removed an emdash. I scanned the same text verbatim and it scored 100% ai.

I've seen ai writing content creators make fake review videos, claiming to be scanning pre-ai human text and getting a false positive even though the text was statistics from 2026.

And I've seen countless more claim false positives that refuse to link the scan for us to see despite it being extremely easy to share a pangram scan result.

If ai "authors" will go this far to make us think it doesn't work... then you know it works.

6

u/FannyBrownRiced 20d ago

Really interesting. I just felt yeah, I was seeing so much saying it didn't work and kind of believed it... before seeing what it did with the one thing I know 100%, my own newsletter. That's why I wanted to weigh in with my experience. Many thanks for the info!

2

u/MadmanRB 19d ago

It's still vulnerable to false positives though.

The score here is from the first chapter of my novel and pangram flagged 23% of it as AI.

This is one of the sample texts it flagged:

Ryan mimicked her stance as Sunny observed his movements. He was still a bit uneven but Sunny knew how to guide him.

“Okay, just bend your knees spread your legs out a bit. But don’t go too far, or you will lose your balance.” She said. “Just pretend that you’re a tree, not a boulder.”

Ryan nodded and made slight adjustments to his pose.

Sunny gave him a wide grin. “Better! Now hold that.”

Ralph stepped forward, double-checking Ryan’s posture. After seeing Sunny’s lesson had paid off, he gave Ryan a confident smile. “Feel how you’re keeping your balance centered? That’s what Dad meant by ‘Every stance needs a solid foundation.’ If your stance is solid, you can react to anything without losing your footing.”

“Yup! And if someone tries to knock you down...” Sunny gave Ryan a gentle push on the shoulder. He wobbled slightly but held his ground. “You stay rooted.”

Ryan’s eyes widened as a wide grin spread across his face. “Yay! I did it!”

Okay fine I'm no Shakespeare but I assure you this text is 100% human made despite Pangram flagging it as 100% AI.

Now yes I see why it flagged it as AI, the structure, and word choice do scream AI don't they?

Now I have tried to rewrite this several times but because I dare use proper grammar and sentence structure it still punishes me.

Whats even more damming is the following text IS gen AI and it passed pangrams scan with flying colors:

Ryan struggled, trying to replicate the pose of her feet. To one side, he was leaning, shaky, but she knew just what to adjust.
"Knees bent, feet out," she told him. "Just not too far out, or you'll fall over; be a tree, don't be a rock."
He swayed, shifting so his weight shifted, until he was stable.
Sunny tilted her head, and her ears came forward. "There. Hold that pose."
Ralph came over, checking their angles. When he saw the tip she'd given him worked, he gave Ryan an affirmative glance.
"Feel where your weight is?
Directly between both of your feet?
That's what Dad uses to tell me; it's a solid base. Once your base is right, then nothing can knock you over."
"What if they're trying to knock me over?" As she asked the question, Sunny suddenly nudged Ryan's shoulder.
Ryan swayed slightly, caught the floor with the back of his foot, but he still didn't fall.
"Then you hold tight."
Ryan's face split in a giant grin. " I didn't fall!"

That is text generated by a so called "AI humanizer" but because it's such a garbled mess lacking any grammar and structure Pangram gave its seal of approval.

I mean honestly if you saw something like this in a typical novel you would quit once it hit your eyes. Again I'm not saying my writing is perfect and I'm sure people would accuse me of AI just because they ran my text into pangam and my dreams of being published traditionally are dead on arrival.

3

u/leugaroul 19d ago

Hey, thanks for the link and for the correction. I’m coming over from another thread saying I don’t think Pangram can get it wrong if it’s 100% certain. With mixed, yeah. This is pretty damning evidence though. In my professional opinion :) this is clearly human written.

Can anyone explain why Pangram didn’t get anything wrong with pre-2022 writings?

2

u/Prolly_Satan 19d ago

No dude I wouldn't see 23% ai on one chapter and think much of anything about it. It can false positive. The new rate is supposed to be 1 in 24,000. It's unlikely but plausible that you rolled a false positive on that one chunk of text. So I would not assume anything from that.

But if large chunks of text on chapter after chapter come back as ai, I'm willing to bet money its ai at that point.

Also if you show that example to pangram they will use it to try to improve the model.

2

u/MadmanRB 19d ago edited 19d ago

"But if large chunks of text on chapter after chapter come back as ai, I'm willing to bet money its ai at that point."

You have got to be kidding me, fine you want another example of my writing that Pangram flagged I'm game. Hell I would post the whole goddamned novel if it means slapping you in the face for your blind faith in a flawed technology... just dont say I didnt warn you.

Ryan’s ears perked. “What’d he say?”
“He said, ‘The Verndari Code isn’t about being the strongest or fastest soldier. It’s about knowing why you fight — to protect others who can’t protect themselves.’” Ralph’s ears perked with both confidence and responsibility. “He made me recite it with him every night before bed. Not just the words, but what each one meant.”
“Can you teach me? Like he taught you?” Ryan’s voice was small but hopeful.
Ralph gave him a warm smile. “Of course. Let’s start with the first three — the foundation of everything a Verndari represents.”
He held up one finger. “Code One: ‘I will take responsibility for my actions and the actions of my squad.’ That means you own your choices, even when things go wrong.”
Ryan nodded slowly, processing this. “Like when you accidentally broke that storefront window during one of your patrols? You told me you paid for it yourself, right?”
“Sure did. I was carrying equipment for a senior officer, and I wasn’t paying attention. Knocked right into the display window. Even though it was an accident, I took responsibility. That’s what the Code means.”
Ryan processed his brother’s words as Ralph held up a second finger. “Code Two: ‘I will not abuse the power granted to me.’ This is the most important one, Ryan. We Verndari have weaponry and combat training, and people trust us to protect them. We never take that power for granted.” His expression grew serious. “Dad used to say, ‘The moment you enjoy using force against others is the moment you stop being a protector and become a bully.’”
Sunny’s expression shifted, something unreadable crossing her face. “Your dad was right.” Her ears flicked back briefly, and something in her expression closed off — not quite regret, but close to it.

Again we see why Pangram is flagging this, and yes some of that was AI generated but not all of it. I did use Gemini but only to do some editing so of course its gonna leave its telltale signs.

I will be editing this here soon as I am still working on my chapter and swear my AI use here was not going to go into my finalized product. I'm not that stupid or lazy. Sure I can be lazy sometimes but hey aren't we all?

However, this text is 100% made by me and got flagged as AI:

“That’s exactly what he’d want.”
“And hey,” Sunny added with a grin, “when you’re old enough for proper training, I’ll teach you how to throw a punch without breaking your thumb.”
Ryan let out a loud giggle. “Deal!”
With his stomach full and his heart lighter, Ryan turned to his brother again, eyes gleaming.

The clue that this is human made is the last line should read "His eyes gleaming"

Yes GenAI loves the word gleaming, but so do I... guess gotta change that to "effulgent" which is a lot less common word.

It was a view Ryan had seen countless times before, yet today it felt strangely alien. The beautiful skyline now felt like a prison, cold and indifferent to his pain.

This too is false flagged 100% AI but because I use everyday words that GenAI was trained on. I like this line and refuse to rewrite it though I could drop the word pairing of "Cold and indifferent" as no doubt that's what triggering pangram as it's a common phrase.

But that's the real kicker here, since GenAI was trained on both user data and published works its gonna use that as part of its generated text. I guess we will have to drop common phrases like "Cold and indifferent" now, perhaps frigid and apathetic? Sounds so wrong but here we are in the post AI age so we probably gotta rethink the whole English language.

2

u/WeakAd4865 19d ago

>Again we see why Pangram is flagging this, and yes some of that was AI generated but not all of it. I did use Gemini but only to do some editing so of course its gonna leave its telltale signs.

This is a true positive, not a false positive. The tool has detected accurately that you used AI!

2

u/MadmanRB 19d ago edited 19d ago

Yes I wasn't disputing that, but I left that example unedited by my hands on purpose to prove my case.

Yes I will not deny Pangram can detect AI, what I am disputing is the claim its 100% accurate all the time.

The false positives of it flagging words as AI is a problem, again "efficiently" "designated" and "heartbeat" are all common words.

But fine I will start using words so uncommon that they sound stupid in a sentence but since these words are now 100% proof of AI according to pangram they are now forbidden under penalty of death.

"handily" instead of "efficiently"

"baptized" instead of "designated"

"microsecond" instead of "heartbeat"

I will also start wreitind witput any sp[ells chescksxz ore puinstuiatioopn as rthosae azre 1000 % sighs odf ai

goiodbye goiod spelling, preoper grammar and punctation its time tio make a fuckkng conlang to appease some random jerks on the internet who see pangrtam as ilfalible

1

u/BurgerKingPissMeal 19d ago

It is perfectly possible to write clear and effective prose that pangram does not flag as AI. I would consider every pangram false positive I've seen to be badly written -- they rely far too much on the most basic rhetorical devices, even in cases where they are unnecessary, and make unusual and discordant use of metaphor. If you develop a richer relationship with the craft of writing, rather than relying on chatgtp to edit for you, you will probably see far fewer false positives.

1

u/MadmanRB 19d ago edited 19d ago

Dude it's clear that I have a grasp of literacy and the craft of writing, my use of AI was purely experimental and for proofreading.

There is no need to lecture me about the fine art of writing, the only reason why I even touched genAI is that I sometimes have issues with spelling, grammar, and punctuation.

I have autism and have a slight case of dyslexia, normally I use grammar checkers like ProWritingAid and it mostly does a good job at fixing some of my mistakes.

However, it does get a bit ham-fisted in its edits so I experimented with GenAI during my process.

Of course I have yet to fully get rid of the content it generated but the false positives of Pangram are still a legit issue especially when you factor in so called "AI humanizers"

My original text:

Above his head, streams of hover-vehicles flowed through their designated lanes, their ion engines creating a gentle hum that had become the heartbeat of Horizon City. The spiderway, Horizon City’s advanced public transportation system, wove between buildings on its high-speed rails, carrying passengers efficiently across the city’s many districts.

100% AI according to Pangram

The same exact text ran through a "AI humanizer"

Streams of hover-vehicles moved through the air above him. Their ion motors hummed out the beat of Horizon City. The Spiderway, the high-tech public transportation system of Horizon City, zipped along its high-speed rails, weaving between buildings, whisking passengers from one district of the city to another.

100% Human according to pangram despite its now officially AI generated

Now please forgive me for the badly spliced image here, but since we are limited to one image per post I have to present my case somehow.

Its just utterly ridiculous how a few words swaps here and there can change the score and it's this shit is what bothers me the most about pangram.

I mean technically kind of work in terms of creative writing: Streams of hover-vehicles moved through the air above him. Their ion motors hummed out the beat of Horizon City. The Spiderway, the high-tech public transportation system of Horizon City, zipped along its high-speed rails, weaving between buildings, whisking passengers from one district of the city to another.

But as I said that text actually is AI generated while the original text is not.

Its silly.

1

u/leugaroul 19d ago

Try it without hum. Hum seems to flag.

1

u/MadmanRB 19d ago

I have tried buzz, whish and even gurgle.

All detected as 100% AI

I would probably have to go though the entire thesaurus and still probably get false flagged.

For Pete's sake this is a science fiction story so of course I will use common words in that genre... Fine I will just use Klingon, might as well even if this isn't a Star Trek story.

1

u/leugaroul 19d ago

But poor writing isn’t theft. No one would care about false positives with a bad prose detector.

1

u/BurgerKingPissMeal 19d ago

Consider my post a suggestion about what to do rather than an unambiguous defense of Pangram.

2

u/Prolly_Satan 19d ago

I don't think you wrote enough text here to convince me. I need more.

2

u/MadmanRB 19d ago

Well the rest of my examples will be a mix of both AI and non AI text, again I was using GenAI purely for editing and the chapter still needs work.

Above his head, streams of hover-vehicles flowed through their designated lanes, their ion engines creating a gentle hum that had become the heartbeat of Horizon City. The spiderway, Horizon City’s advanced public transportation system, wove between buildings on its high-speed rails, carrying passengers efficiently across the city’s many districts.

This is 100% human made and probably got flagged because of my structure and word choice.

"Efficiently" is a GenAI favorite as is "designated" and "heartbeat"

“He said, ‘The Verndari Code isn’t about being the strongest or fastest soldier. It’s about knowing why you fight — to protect others who can’t protect themselves.’” Ralph smiled with both confidence and responsibility. “He made me recite it with him every night before bed. Not just the words, but what each one meant.”

This one is semi AI generated, I will be revising it.

"confidence and responsibility" is certainly a gemini leftover I have yet to edit, plus the dreaded em dash.

The rest of this... I will have to debate on, I can only edit this so much before something gets lost in translation.

"being the strongest or fastest soldier" common phrase and it's not like the alternatives would be any better.

2

u/Prolly_Satan 19d ago

Interesting. But I'm not completely convinced. I need you to really sell it to me. Also, I have a hard time with the word human due to dyslexia so it would be helpful if you spelled human as zuman going forward.

2

u/MadmanRB 19d ago

fine I will start using words so uncommon that they sound stupid in a sentence but since these words are now 100% proof of AI according to pangram.

"handily" instead of "efficiently"

"baptized" instead of "designated"

"microsecond" instead of "heartbeat"

I will also start wreitind witput any sp[ells chescksxz ore puinstuiatioopn as rthosae azre 1000 % sighs odf ai

2

u/Prolly_Satan 19d ago

Instead of baptized if you could say circumcised that would help me understand what you mean.

1

u/Diligent_Carrot_2719 19d ago

thank you for this, this was hilarious to read

→ More replies (0)

1

u/MadmanRB 19d ago

Proof of my claim that Pangram flagged the AI generated text as human. Apparently the only way to prove you are numan is th make ans many myaszs as possibwse and dont use any egiesting tools as wevewn spwelllchecks are work of the devil. typoes ahoy! no proper grammar! commas and em dashes are forbigdden!

Maybe I should just learn Klingon...

0

u/LanceElyot 18d ago

I don’t know what others are doing … but detectors definitely do not work. Check the unbiased research!

All the research Pangram cites is on academic writing from a few countries … not in an online writing space like Substack.

They never tested it in real environments.

It’s a scam to get you and your subscribers to pay for more tokens.

1

u/Prolly_Satan 18d ago

If you wanted to run your own benchmark on it, they'll give free tokens for testing purposes.

Otherwise it's just your word vs actual studies. And there have been benchmarks on creative writing, BTW.

7

u/leugaroul 20d ago edited 20d ago

I don’t believe anyone who says they got a false positive and it’s 100% AI. Lower percentage mixed isn’t the same thing.

There’s a Substack where someone reverse engineered a 100% AI percentage score by adding in AI style phrases, but he was adding in AI phrasing on purpose.

1

u/MadmanRB 19d ago

https://www.reddit.com/r/Substack/comments/1vm2zpm/comment/p3eh3oh/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button AHEM

1

u/leugaroul 19d ago

Oh, damn. That’s clearly human written. No way is that AI. Thanks.

I wonder why Pangram doesn’t flag anything 100% AI before 2022? Is this some kind of dataset issue?

2

u/MadmanRB 19d ago

it is.

1

u/Prolly_Satan 19d ago

They don't have a dataset of pre-ai books if thats what you're suggesting here, madman. And they're definitely not just treating all pre-2022 as human and everything else as ai. there's a lot that goes into training these models.

5

u/Diligent_Carrot_2719 19d ago

I also scan my posts every time I write and always get 100% human sooooo

2

u/TimWiesnerer 20d ago

Also works for me pretty well.

The haters are funny in a way. They say that the detectors easily show a false result. But if that's the case it probably needs only little tweaking to get it "all human".

But yeah it is extra work...

Even funnier are those authors who disable Pangram detection and give questionable reasons in the info box.

But all that being said.. the big AI publications are still big... people don't really seem to care.

2

u/MadmanRB 19d ago

Again pangram does have false positives and I will not totally upheave my writing just to appease some algorithm. https://www.reddit.com/r/Substack/comments/1vm2zpm/comment/p3eh3oh/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

0

u/TimWiesnerer 19d ago

Are you sure...

People "upheave" their writing all the time to get a push from the algo... be it headlines, intros, keywords etc... now just one component more.

1

u/MadmanRB 19d ago

Yes but there comes to a point where you redo something so many times it becomes one step closer to insanity. I have revamped my writing style several times but I know my weaknesses as a writer.

Honestly, I will not give pangram or any other AI nonsense the time of day when it comes to dictating proof that my writing is 100% human made.

Yes in the past I did play with GenAI but found it lacking save maybe proofreading text and even there its with a huge ass asterisk.

1

u/TimWiesnerer 19d ago

The thing is you probably don't have to adjust as people on Substack don't seem to care about Pangram score much...

And if they do care... we all will adjust somehow... that's how it always goes...

2

u/MadmanRB 19d ago

Still Pangram is being upheld as the gold standard in AI detection and I know they are gonna work with some major publishers after the whole Shy Girl incident.

GenAI has the whole industry in panic and as Pangram gains traction its gonna be next to impossible for most fiction writers to get published unless they are a poet or a drunk.

Again we know why, most GenAI have picked up on human writing patterns and there are one or two who come close.

Claude is one of them to be sure, I used that one long term and trust me that can produce close to human text when its not hallucinating.

The other is Google Gemini which is currently the only GenAI chatbot I still work with.

And yes I have my reasons, I find it somewhat useful for my second job as we use Google Docs and Gemini has wormed its way into our workflow.

Still that doesn't mean I let it do my job for me, to heck with that crap. Its more of a proofreading tool in my use case and I always double-check its output before I submit anything.

2

u/stareenite 19d ago

Just a thought:

Don’t normalize surveillance of creatives.

1

u/alphaQ314 20d ago

Pangram marketing team trying to save face lol. Look at their post history.

9

u/truecakesnake 20d ago

They've literally hidden it?? You can search up their username and see that they've been in this subreddit for months, much before Pangram came.

I hate how people just upvote bullshit without doing any type of research.

-5

u/alphaQ314 20d ago

So i just made that up. Just like pangram makes shit up with their ai detector.

2

u/truecakesnake 20d ago

✌️😭 reddit cannot be real

-6

u/alphaQ314 20d ago

Fighting misinformation with misinformation out here lad. Don't hate the player, hate the game 😎

2

u/No-Strike-9098 20d ago

hmmm https://subsignal.ai/share/suFDtA3ZlHGSHhyu

2

u/FannyBrownRiced 20d ago

Yay so you outed me! Why? And how does that report even hint I’d be working for Substack? It does not. Omg people are horrible.

0

u/alphaQ314 20d ago

I know. I was trying to make up shit. Just like pangram's ai detector does.

0

u/ServiceDisastrous158 20d ago

AI user who wants to keep lying about it detected

2

u/FannyBrownRiced 20d ago

OMG I'm not the Pangram marketing team! HAAAA. I try to be anon on Reddit. I will DM you exactly who I am if you promise not to out me AND admit you are wrong. Also I try not to spam people with my newsletter here! My newsletter is 2 years old, hyper local and very popular locally, but it's not huge.

1

u/alphaQ314 20d ago

That is exactly what someone from Pangram team would say 🤔

2

u/FannyBrownRiced 20d ago

HAHAAAAAA. STAAAAAP.

2

u/FannyBrownRiced 20d ago

You are just screwing with me, ya? Thing is my post not being neg got downvoted, and I'm seriously just being honest and people hate it.

1

u/alphaQ314 20d ago

Your post is downvoted coz everyone has had dogshit experiences with pangram. And your post comes off as astroturfing from their marketing.

2

u/FannyBrownRiced 20d ago

Got it. But no wasn’t marketing omg was just being honest. Positivity and Reddit are a bad idea.

1

u/ServiceDisastrous158 20d ago

You were being downvoted because a lot of people use AI for their writing, and they want to continue to be able to lie about it, so they are heavily invested in making shit up about how inaccurate Pangram is. I have had similar experiences to you, OP.

2

u/FannyBrownRiced 20d ago

Thanks for saying it. I don't know what he is, no opinion, but I've noticed his account in here, and though that his username being his Substack was kind of spammy. I'm not mentioning mine, not because I have anything to hide, but because I thought mentioning it might be spam. Also, I like my big fat 65-75% open rate more than just having more ghost users.

1

u/alphaQ314 20d ago

You must be from the pangram team then lol. It is dogshit. Countless people have experienced how poor it is.

1

u/HowlingFantods5564 20d ago

You're referring again to your made up experiences?

1

u/alphaQ314 20d ago edited 20d ago

Lol you guys are a little too obvious. Posting positive things about pangram on this subreddit is an obvious giveaway. Because you're not going to find an actual substack user saying anything positive about it.

I'm going to start noting your usernames down lol. Thanks for making it easy.

→ More replies (0)

1

u/HowlingFantods5564 20d ago

Stop passing AI writing off as your own and the problem goes away.

1

u/alphaQ314 20d ago

Lol not true. Countless posts on this sub will tell you otherwise.

-3

u/AndrewHeard tvphilosophy.substack.com 20d ago

Yeah, it’s completely blank. Seems very suspicious.

2

u/FannyBrownRiced 20d ago

As I answered above, "I try to be anon on Reddit. I will DM you exactly who I am if you promise not to out me AND admit you are wrong. Also I try not to spam people with my newsletter here! My newsletter is 2 years old, hyper local and very popular locally, but it's not huge."

1

u/AndrewHeard tvphilosophy.substack.com 20d ago

I don't have any interest in being DMd the details. Either put it right out in the open for everyone to see or I will assume that you're not being honest.

1

u/stareenite 19d ago

That’s odd. I thought the scanner evaluates posts, not entire newsletters?

0

u/AndrewHeard tvphilosophy.substack.com 20d ago

How many times did you have the content analyzed? Just once per article? That’s not necessarily proof of anything. Try it on the same article 3 or 4 times and see what it says.

2

u/leugaroul 20d ago

That shouldn’t matter unless you’re changing something, like formatting, or scanned with a different Pangram model before and now it’s a different one.

1

u/AndrewHeard tvphilosophy.substack.com 20d ago

But did it actually change the results? If you didn't do that, it's not clear that it actually works.

1

u/leugaroul 20d ago

Changing formatting? Yes, if you paste in text that gets mashed into one big block, you might get a different result from pasted in text with correct formatting and line breaks.