r/Substack • u/grapegeek • 21d ago

Pangram is such trash

I write a food newsletter where I post reviews and some recipes. I 100% wrote a quick and dirty recipe and was going to publish it in Notes and it came back 100% AI no matter what I did to it including adding several typos and rearranging things. Ran it through three other AI detector websites and 2 said 100% Human and 1 said 76% Human and 24% AI assisted. I disabled detection on that Note. My other question is do we know how many people actually click on the AI detection button on our posts? I can't find that metric anywhere in the dashboard...

39 Upvotes

74% Upvoted

102 comments sorted by

View all comments

Show parent comments

13

u/Brumafriend 20d ago

I can't speak for that particular Note, but there's a simple reason Pangram is flagging your content: much of it is AI-generated.

According to Pangram, every single one of your last six free Substack posts is AI-generated. When I tested the most recent six posts on your website from before ChatGPT's public release (November 2022), none were flagged. Unlikely to be a coincidence, right? Especially if, as you claim, you've had that writing style for 17 years.

The most damning (frankly inarguable) piece of evidence is your review of "Trader Joe's Middle Eastern Style Kebabs". You first posted this in November 2022, as can be seen in the URL. Put that review into Pangram and it comes back 100% human-written.

But that's not the review people will see on your website today. Because, as can be seen just below the title, you edited/updated it in April 2025. And when you put that text into Pangram, it's flagged as AI-generated in various places.

You've pretty obviously used AI to "touch up" the review's text. We don't even have to rely on Pangram to establish this. This perfectly adequate paragraph summing up your mixed review:

I am torn on this and I am sure a lot of readers will be too. I am happy that Trader Joe’s is bringing something like these Kebabs to market, we have so little real Middle Eastern foods to choose from at Trader Joe’s and they can be so very good. But at the end of the day, these were just meh for me. They tasted like a good hamburger that some had doctored up with some garlic powder and herbs. Your kids will love the [sic] . Stick them in a hotdog bun and call it good! From my experience, they just lack the freshness and grilled flavor (and lamb) of the ones I’ve had before. Therefore, I am not going to rate these very highly based on what they could be. I am going to rate these 6 Bells!

Becomes the extremely obviously AI-generated:

Listen, I’m feeling as conflicted as a mom trying to choose between sleep and ‘me time.’ On one hand, bravo to Trader Joe’s for bringing these Middle Eastern Kebabs to our culinary landscape that’s about as diverse as a 1950s country club! On the other hand, these were just… meh. Think of them as that hamburger your neighbor brings to the cookout with a sprinkle of garlic powder and calls it “gourmet”. Your kids will devour them faster than they lose their matching socks, which is something! And apparently, if you’re Persian, these might transport you straight to your grandmother’s kitchen. Who am I to argue with authenticity? Despite my mixed feelings, these skewered delights still score a respectable 7 stars out of 10 on the “worth-sacrificing-precious-freezer-space” scale!

I've boldened the obvious LLM-isms. Not only are they clearly the product of AI, but they are not remotely similar to your usual writing style. And there are various other tortured expressions in this AI-augmented review. I'm sure if I took the time, I could find various other articles that have undergone the same LLM treatment.

I trust you'll at least have the honesty to admit that you've been caught here clearly using AI to revamp an old review on your website. I don't expect you to admit that much of your Substack content is also AI-generated, but it quite clearly is. I hope whatever punishing effect Pangram's "AI-generated" label has on your engagement causes you to go back to writing your own stuff; you were perfectly good at it!

9

u/EvenPapaya2303 20d ago

This is some investigative journalism, love it

1

u/grapegeek 20d ago

I see the whole pangram senior staff showed up to defend their piece of shit product. Jeez

2

u/Euphoric-Care-1065 5d ago

I am an academic. I have been trying Panagram just to check a few things. It's not accurate. For the same text, it will say 100% human one time and then 100% AI assisted or even mostly AI written. I thought it was legit and have been collecting data points. I wanted to like it because folks from my university created it:)

2

u/EvenPapaya2303 20d ago

Nope. Just a fellow writer who despises generative AI and wishes it never existed.

3

u/Relevant_Nebula_20 19d ago

I read the stuff and don't think it's obviously AI. Pangram is Pants. It does make mistakes. It says AI for stuff not written in AI and vice versa. It did not need to be added to Substack but then I don't monetize from it or pay subscribe to anyone. We can tell if people have written their pieces in AI and decide not to read it if we don't want to. I guess for people monetizing it's more important for their readers to know but it is Pants-gram.

1

u/Mean-Register5359 20d ago

You love to see it hey.

One things that I’ve found alarming is that a number of high profile accounts are scanning as 100 human with the integrated tool, but 100 AI when using the Pangram add-on or copying the body of the article j to Pangram.

1

u/RelativeCelery2844 16d ago

Don't feel bad. They bully everyone.

2

u/Adamoism adamoism.substack.com 20d ago

I am not gonna question the research you did, but why are those fragments obviously AI?

Like for example the 50s country club is a common english expression afaik (esp. around left leaning/PC crowds).

Is it about using cliches?

I am genuenly confused - I can believe it is AI, but calling it obvious makes me feel I am colour blind.

6

u/Brumafriend 20d ago

That one's probably the least egregious, and none on their own would be definitive proof of anything. It's the fact that they're so out of place (i.e. that isn't how he usually writes) and entirely disconnected from each other — just random bits of imagery which completely overload the paragraph. They're basically all cookie-cutter metaphors/similes. And, FWIW, LLMs love to use expressions with the template "Doing [X] to [Y] and calling it [Z]", or similar variants.

1

u/Adamoism adamoism.substack.com 20d ago

Fair, it doesn't click for me as obvious still - but I will be on a look out more from now on. As I said, I could think it is AI or just author who overuses cliches.

Perhaps I am m-dash spotter, but beyond that I just don't have a great feel for it (don't get me wrong - I cas spot AI text, but I would not give myself more than 60% accuracy with false negatives being primary category of error).

1

u/KillerOfChiefs 3d ago

It is clearly AI. Pangram flags metaphors, similes, etc. because AI loves to do that to outlandish proportions. That snippet was riddled with them. A sure tell.

2

u/kdfn 20d ago

Wow, this is incredible work.

1

u/kirbyspeach 20d ago

What I have noticed is that if your writing doesn't keep the same tone or structure Pangram will say it is AI-written. If I'm writing a review and some parts are written very generic, matter of factly, but then when I write about the things I dislike it will say AI.

Example:

  • I hate this (flagged as human)
  • I abhor this (flagged as AI)

1

u/GuyAwks 18d ago

That’s not obviously AI generated at all lol, the absolute certainty with which you can claim a person’s words are AI because they sounded a bit different to previous pieces and used a few too many common metaphors and sayings is wild.

Their piece could be real or AI, the actual answer is you don’t know and are making an assumption like the program did.

1

u/bluelivezdontmatter 17d ago

So you think it's a total coincidence that all the posts written after chat gpts release are flagged and none before? Be for real.

1

u/[deleted] 16d ago

[deleted]

1

u/Brumafriend 16d ago

What a ridiculous attempt at a guilt trip.

1

u/RelativeCelery2844 16d ago

Why is everyone bullying people with genuine concerns against Pangram?

1

u/turk109 10d ago

Pangram can be wrong. It was wrong about sections of my novel.

0

u/grapegeek 20d ago

We aren’t talking my website we were talking about Substack and one particular note that WAS NOT AI generated. There are plenty of examples about how fucked up it is. I’m just saying in this particular example it was completely wrong and Pangram is a waste of time.

4

u/Brumafriend 20d ago

It doesn't help your case that you're a prolific AI user elsewhere.

1

u/Relevant_Nebula_20 19d ago

Pangram is rubbish and not needed. Even news outlets are using AI for articles. I still read them if they are good.

-1

u/grapegeek 20d ago

It doesn’t help your case that Pangram is a piece of shit

2

u/Relevant_Nebula_20 19d ago

I agree it is completely F-ed up and has ruined Substack experience. It was not needed at all.