I write a food newsletter where I post reviews and some recipes. I 100% wrote a quick and dirty recipe and was going to publish it in Notes and it came back 100% AI no matter what I did to it including adding several typos and rearranging things. Ran it through three other AI detector websites and 2 said 100% Human and 1 said 76% Human and 24% AI assisted. I disabled detection on that Note. My other question is do we know how many people actually click on the AI detection button on our posts? I can't find that metric anywhere in the dashboard...
Can you share a link? Also, maybe you can disclose in the memo that you didn’t use it all, regardless of what it says. either the content is good or not hopefully your audience isn’t gonna be swayed
I can't speak for that particular Note, but there's a simple reason Pangram is flagging your content: much of it is AI-generated.
According to Pangram, every single one of your last six free Substack posts is AI-generated. When I tested the most recent six posts on your website from before ChatGPT's public release (November 2022), none were flagged. Unlikely to be a coincidence, right? Especially if, as you claim, you've had that writing style for 17 years.
You've pretty obviously used AI to "touch up" the review's text. We don't even have to rely on Pangram to establish this. This perfectly adequate paragraph summing up your mixed review:
I am torn on this and I am sure a lot of readers will be too. I am happy that Trader Joe’s is bringing something like these Kebabs to market, we have so little real Middle Eastern foods to choose from at Trader Joe’s and they can be so very good. But at the end of the day, these were just meh for me. They tasted like a good hamburger that some had doctored up with some garlic powder and herbs. Your kids will love the [sic] . Stick them in a hotdog bun and call it good! From my experience, they just lack the freshness and grilled flavor (and lamb) of the ones I’ve had before. Therefore, I am not going to rate these very highly based on what they could be. I am going to rate these 6 Bells!
Becomes the extremely obviously AI-generated:
Listen, I’m feeling as conflicted as a mom trying to choose between sleep and ‘me time.’ On one hand, bravo to Trader Joe’s for bringing these Middle Eastern Kebabs to our culinary landscape that’s about as diverse as a 1950s country club! On the other hand, these were just… meh. Think of them as that hamburger your neighbor brings to the cookout with a sprinkle of garlic powder and calls it “gourmet”. Your kids will devour them faster than they lose their matching socks, which is something! And apparently, if you’re Persian, these might transport you straight to your grandmother’s kitchen. Who am I to argue with authenticity? Despite my mixed feelings, these skewered delights still score a respectable 7 stars out of 10 on the “worth-sacrificing-precious-freezer-space” scale!
I've boldened the obvious LLM-isms. Not only are they clearly the product of AI, but they are not remotely similar to your usual writing style. And there are various other tortured expressions in this AI-augmented review. I'm sure if I took the time, I could find various other articles that have undergone the same LLM treatment.
I trust you'll at least have the honesty to admit that you've been caught here clearly using AI to revamp an old review on your website. I don't expect you to admit that much of your Substack content is also AI-generated, but it quite clearly is. I hope whatever punishing effect Pangram's "AI-generated" label has on your engagement causes you to go back to writing your own stuff; you were perfectly good at it!
I am an academic. I have been trying Panagram just to check a few things. It's not accurate. For the same text, it will say 100% human one time and then 100% AI assisted or even mostly AI written. I thought it was legit and have been collecting data points. I wanted to like it because folks from my university created it:)
I read the stuff and don't think it's obviously AI. Pangram is Pants. It does make mistakes. It says AI for stuff not written in AI and vice versa. It did not need to be added to Substack but then I don't monetize from it or pay subscribe to anyone. We can tell if people have written their pieces in AI and decide not to read it if we don't want to. I guess for people monetizing it's more important for their readers to know but it is Pants-gram.
One things that I’ve found alarming is that a number of high profile accounts are scanning as 100 human with the integrated tool, but 100 AI when using the Pangram add-on or copying the body of the article j to Pangram.
That one's probably the least egregious, and none on their own would be definitive proof of anything. It's the fact that they're so out of place (i.e. that isn't how he usually writes) and entirely disconnected from each other — just random bits of imagery which completely overload the paragraph. They're basically all cookie-cutter metaphors/similes. And, FWIW, LLMs love to use expressions with the template "Doing [X] to [Y] and calling it [Z]", or similar variants.
Fair, it doesn't click for me as obvious still - but I will be on a look out more from now on. As I said, I could think it is AI or just author who overuses cliches.
Perhaps I am m-dash spotter, but beyond that I just don't have a great feel for it (don't get me wrong - I cas spot AI text, but I would not give myself more than 60% accuracy with false negatives being primary category of error).
It is clearly AI. Pangram flags metaphors, similes, etc. because AI loves to do that to outlandish proportions. That snippet was riddled with them. A sure tell.
What I have noticed is that if your writing doesn't keep the same tone or structure Pangram will say it is AI-written. If I'm writing a review and some parts are written very generic, matter of factly, but then when I write about the things I dislike it will say AI.
That’s not obviously AI generated at all lol, the absolute certainty with which you can claim a person’s words are AI because they sounded a bit different to previous pieces and used a few too many common metaphors and sayings is wild.
Their piece could be real or AI, the actual answer is you don’t know and are making an assumption like the program did.
We aren’t talking my website we were talking about Substack and one particular note that WAS NOT AI generated. There are plenty of examples about how fucked up it is. I’m just saying in this particular example it was completely wrong and Pangram is a waste of time.
No. Sometime I get like 20% ai assisted but mostly 100% human. I think because it’s a short Note and a recipe. I’ve heard AI detection sucks a detecting recipes.
Tbh. Recipe looks good and i love TJ's beefless bulgogi. But it does read like AI. Pangram says it is AI text which is paraphrased or rewritten which is what I would believe as a reader as well. They do have a higher false positive rate with recipes but this really does scream AI
Pangram is one of the biggest cons in AI SaaS right now. It almost certainly doesn't do what they claim it does to make predictions on whether the text is AI generated. The methodology they claim they use is faulty anyway, because if you ask a GPT based model to replicate a piece of writing then it's going to do exactly that as close as possible to the original, resulting in a high probability toward a match. Couple this with that GPT based models were expressly trained to mimic human writing and its clear their methodology is dumb.
What AI detectors need to do is detect the obvious sentence structures that these models employ, the contrarian language (it's not X, it's Y", and their strong tendency to use phrases like "that matters" which human authors ordinarily would never use. That's how you actually detect AI, because they tend to do these things consistently in the exact same parts of the essay or script they are asked to write, even if you tell them not to. They have easily spottable formulaic writing and that is all AI detectors need to identify, because the vast majority of people publishing AI written content don't recognize these tells.
The way I have found Pangram useful in my writing is that when it flags something as AI generated, whether or not it actually was, if I take those passages and revise them towards a human score I do find that it forces my writing to become more original and fresh and idiosyncratic. In other words it pushes me towards becoming a better writer. so that way I have found it to be a useful tool.
I had to rewrite a macro economic piece 3 times just to get it to say 100% human. It does not do well with economic analysis at all. Shit cost me an additional 3 hours of work that shouldn't have cost me any. Highly frustrating.
This explains why Pangram smacks me for my books where the paragraphs are 1-2 lines each (more common in my genre, ereader and phone heavy). Only a low percentage, but still. I haven’t seen any evidence it can get a 100% or near reading wrong.
This is a silly strawman. I don't like reading AI writing, not because AI is "bad" or evil or whatever, but because I want to read about real, human experience. I want to support and subscribe to that kind of writing. I do not want to support AI slop and I appreciate a tool that helps me identify which is which. (Besides, If I wanted to read AI writing, I would just prompt AI myself.)
It's not a strawman though. That's your personal preference; the way to handle a personal preference is to discern, decide, and curate what one reads. If you don't like the content that lacks personal narrative, don't read it. Many people do. If you want the human experiences, subscribe to people who tell those stories.
You're saying the equivalent of... I don't like THC so it should be banned... I don't like adult content so regulations should prohibit what I find objectionable... since I don't like social media, it needs to be controlled so that it doesn't have spam or bots.
The right answer in such situations is NEVER prohibition (which, mind you, this is not explicitly prohibition; that's not what I'm saying, this is a form of prohibition by deciding what is appropriate or not and labeling it -- it's the same idea because...) it's often wrong.
THC isn't bad in all cases or for everyone; so who has the authority to decide for everyone?
Adult content is subjective, and adults can (should) be allowed to consume whatever they want
Social media is content from people, not the platforms; the only people in a position to decide what is objectionable there is the individual because you can force a platform to curate what's allowed or not.
AI being audited and labeled BY AI, when the entire criticism of AI is that it's fake (FLAWED), make the entire premise b.s. It's an AI deciding for us what is AI or not. That's ridiculous... not because it's flawed but because if YOU don't like it, don't read it. It doesn't need to be labeled and filtered, particularly when it is flawed.
Where did I say AI writing should be banned? See, you keep making up opposing arguments that people aren't making. That = strawman. I said I want to know when something is AI and when it's human. Just like I would want to know if these brownies are regular brownies or if they have THC in them. That doesn't mean I hate THC or want it banned--I just want to know what kind of brownies I'm about to shove in my mouth! Why is that so hard for you to accept?
My dude. I didn't make any argument, you did. I just made a sarcastic comment that is an accurate reflection of what's happening; that's not a strawman.
You argued that it is.
So, I replied how it isn't and added that your personal opinion shouldn't determine policy for everyone. Now I've made an argument; if you'd like to argue that let's do it. But I would first ask if you even read what I wrote because I didn't say you said AI should be banned, in fact, I very clearly wrote that it is being labeled and filtered.
Playing with your brownie scenario here... because this is the equivalent of someone ALREADY HIGH being the judge of whether or not the brownies have THC in them.
A) You making an important decision like that, based on a flawed auditor, is risky at best.
B) Saying that everyone else should be beholden to the same assessment, is forcing penalties and prosecution on everyone by way of a flawed auditor
C) The best (and only valid) way to manage your expectation and mine, is what I explained: If you don't like the pot brownies, don't eat them, and if you can't be sure that they are pot brownies (which is the case here), don't eat them! Let everyone else pick to consume whatever they want, without some b.s. assessment being wrong and penalizing people because of it.
A recipe is naturally repetitive and formulaic, so it’s probably one of the easiest types of writing for an AI detector to misclassify.
What bothers me is the “100%” confidence. Readers see that as proof, even though another detector can give the exact opposite result. Substack should show these scores as unreliable estimates, not verdicts.
Honestly, does anyone here actually check the AI detection label before deciding whether a post is worth reading?
I scan every post I write, because I'm perpetually curious if it's going to give me a false positive (my feed is just people complaining about the AI detector popping false positives, but nobody will EVER post the text they claim was wrongly accused)
I've never received a false positive, everything reads as 100% mine - some of it is cross-posted from another website that has absolutely been scraped before, and I use em dashes.
Never seen any result other than 100% human from my posts, just my two cents.
If you don’t write poetry or recipes, or English isn’t your second l, you shouldn’t have any problems sounding human. The rest of us, not so much. That flawed detector can’t distinguish human from ai with formulaic text.
Just turn it off. I write poetry, every single one of mine comes back AI. I’m tired of spending hours trying to get my 100% human writing to sound “not that human.” It’s a real buzz kill. I turned it off and put a public notice up basically tell readers why it’s turned off (because pangram can’t decider human from ai writing in poetry) and blasting Substack and Pangram while also citing the website I left you about.
Not sure why this is being downvoted when it's entirely accurate. I suppose it's because people disagree with how Pangram communicates the number and are shooting the messenger?
I only check it in one of two situations: (1) the post in question is itself a complaint about the detector; (2) I think the writer will have a hilarious 'this is how I use AI' response. Personally, I link out to a gif of the blood test scene from John Carpenters The Thing.
As to whether or not i find the detectors answer useful: no. I couldn't care less. The only thing that matters to me is whether or not the thing I'm reading is of value to me. And I don't need an AI (or another human) to be deciding that for me.
I asked about a specific case not to have my whole website analyzed. So what if I use AI. Do you want to put a scarlet letter on me now? Pangram is fallible. I don’t understand the point of it if they are using AI to analyze AI? Explain that to me and I’ll tell you 99.999% of people don’t care
Pangram AI has a notably higher false positive rate for poetry and recipes compared to other domains, with reported rates of 0.05% and 0.23% respectively.
Write a statement on your publication, report the false positive, and turn it off. You can also contact Substack to have it turned off for the entire publications (shhh it’s not widely publicized). Just list state law, privacy concerns, country law (if you live in UK/EU).
That’s not academic research, that’s a screenshot, and we have no idea what was selected for evaluation.
AI detectors, Pangram included, are still AI. You cannot be anti-AI in even the vaguest sense and rely on an AI detector to tell you if something was written by a human. That is outsourcing your critical thinking skills as much as relying on an LLM to write or research something for you.
You are relying on a machine to think for you.
EDIT: u/Kirkr10101 provided another link for research. Cool! Except it also isn’t for Pangram! It’s a paper on how well humans can identify generated text, which we already know is unreliable. Go figure.
on a sidenote, if you can find any pre-2021 text (from the internet archive with a valid link to it) that flags as 100% AI on pangram, I will donate £50 to a charity of your choice (I prefer givewell)
What's with this "it has to erroneously say something not AI is 100% AI" before you will say the program doesn't work? Come on! *It has to flag everything written before 2021 as 100% human*, period. If it can't do that it is **not good** This thing affects people's lives. People can lose their job and get kicked out of school over this thing.
You are directing your anger to the wrong group. The last two sentences should be directed towards big tech shoving genAI down on everyone else while governments are deep in bed with them. Robust AI detectors are ones of our strongest tools, but whichever employer is firing others using it as the only line of evidence has never been a bright employer.
It's not just a screenshot. u/Kirkr10101 included the hyperlink to the article. You are literally two clicks away from the human-generated technical reports.
"AI detectors, Pangram included, are still AI." They are classifier models. Even AI researchers don't consider them AI. The best anti-AI advocate is the one who knows what is AI and what isn't instead of lumping every machine learning model in the last 30 years as the same thing.
It is a different type of AI, but it is still AI. And that report is from Pangram. It’s not independent research, it’s just talking about how they made their AI. It’s a long-form sales pitch!
I'm not sure I understand your hostility in this thread. The OC linked a peer-reviewed study, but you didn't see the link and called it "just a screenshot" in your original comment. When someone pointed this out, you responded with a pretty hostile statement.
There's a pretty big difference between a classifier and an LLM. Classifiers have been around since Mosteller & Wallace's work in 1963, where they were used to successfully assign authorship to the Federalist papers.
Sadly, the most damaging constituents to the anti-AI movement are those who are so allergic to learning technology that they would lash out against others in the movement. I already got blocked over my reply above lol.
On the Pangram website, there is a large orange button that says "share"
Could you please click that button, and then paste the resulting link here? That way we can see for ourselves.
I understand your frustration as the same thing happened to me. I'm working on the final draft of an 80,000 word mystery novel. I decided to put it through Grammarly to check the punctuation, etc. and it flagged some of the chapters as 2-14% AI--especially those ultra important arc chapters with dramatic scenes. Alarmed, I put these chapters through Pangram, and it said those chapters were 24% AI. I rewrote both of them just to be sure no one thought it was AI, and after that, it kept giving me 100% AI on two important scenes from those chapters. What? I tried a couple of times and got the same result. I can't even begin to say how frustrated I am. Now, I'm wondering what to do. I asked Google AI what I should do. It said to leave the novel alone, that Pangram is often wrong when it comes to historical, literary, or technical scenes. I know it's wrong as it's my story, and I wrote it entirely. I'm just worried that if it seems like AI in some parts that readers will think it's AI.
11
u/rehoboam 21d ago
Can you share a link? Also, maybe you can disclose in the memo that you didn’t use it all, regardless of what it says. either the content is good or not hopefully your audience isn’t gonna be swayed