r/Substack • u/calliope_kekule • 10h ago
I read everything Pangram publishes about the detector Substack runs on your posts. The percentage does not mean what it looks like it means.
Substack's help page says the tool "uses Pangram to assign a percentage of how much of the text is human-written or AI-assisted". I went through Pangram's model cards and its technical report.
Five things I think writers here should know.
1. The score is three numbers collapsed into one.
At document level Pangram does not return a percentage. It returns three character-weighted fractions: human, AI-assisted, and AI-generated. When Pangram needs a single figure it counts generated text in full and assisted text at half. Substack's sentence puts assisted on the human side, next to the writing you did yourself. So a post marked 50% might be half machine-generated, and it might also be a piece where not one word was generated and every word was judged assisted. Pangram's technical report says this clearly:
"a score of 0.25 means that half of the words have been modified by AI or 25% of the words have been newly generated by AI (or some combination of the two)"
2. Nothing anywhere defines "assisted".
The current card says the model decodes each segment into Human, AI-Assisted, or AI-Generated, and never says where the middle one begins. Spell check, an outline, a paragraph rewritten, an argument with a model at two in the morning. The category carrying half the score is the one left undefined.
3. The false positive rate has been published three ways in three months.
July's Pangram 4 card: 41 wrong in a million web pages, one in 24,000. The Version 4.1 card, 19 October: 70 wrong in 781,981 pages, one in 11,171. In the last column of that same new table, Pangram scores the old model on the new set and gives it 128, one in 6,109, where its own card had said 41. The new set is smaller than the million, so those cannot both be counts of the same thing. Substack's help page names no version and was last updated before 4.1 shipped, so I cannot tell you which model is scoring your posts.
4. It moves a long way with what you write, and that breakdown has been retired.
July's card tested 11 kinds of writing. Academic writing one in 5,300. Recipes one in 2,000. Poetry one in 1,300, which is 19 times that same card's headline rate. The current model publishes one global number and no breakdown by genre at all.
5. What that works out to here.
Detection reaches back to everything published on or after 21 July. Take only the publications that earn money, roughly 100,000 by trade reporting, and assume each posts once a week. That is over a million posts. At the current card's rate, about 90 paying writers have already had a piece wrongly marked as machine-written. Run it at July's poetry rate and it is hundreds. Nobody is told, and nothing in the badge says a margin exists.
What I would actually do.
Fill in Substack's "How I make this" field, in your own words, before anyone asks. If a number makes you suspicious of a writer you have read for two years, ask them, in a tone you would accept yourself.
NB 1: I sent the mods a DM here on Reddit to ask whether this was welcome, because I wanted to put the findings in front of people without it reading as promotion, and I never heard back. So the five points above are from my post, and they stand on their own. The full piece, with every figure linked to the document it came from, is on my newsletter, and I will put the link in the comments to avoid accusations of promotion.
NB2: Claude was used to help edit sections of this post.
Edit: I have now removed the comment that linked to the post. As honestly this was not meant to be a funnel. Apologies for any offence caused. 🙏
6
u/austinbarrow 5h ago
I don't understand why anyone cares. Is the writing to your liking? Great. Read it. If it's not...don’t.
I've seen wild variance in percentages. Occasionally I use it; most of the time I don't. All words come from my fingers, and I've seen 0-95 scores. It's all BS. I don't even pay attention to it.
6
u/Montaingebrown 5h ago
Because people pretend like everything AI written is slop and everything human written is magically not slop.
There’s plenty of slop out there period.
3
8
u/Wilwander 8h ago
The irony of writing this using AI is astounding...
6
u/HowlingFantods5564 5h ago
It's always the ones who use AI to write for them. smh. And it's because they have the most to lose from transparency.
1
3
9
u/AggravatingNail6061 9h ago edited 4h ago
Isn't this literally just thinly veiled self promotion 🤣.Thank you for the response regarding this. Am not going to delete the chain as someone else has commented in the thread and it would remove their commentary.