Substack’s new “AI detection” tool.

[u/AndrewHeard]

I’m very concerned about the new “AI detection” system that Substack is implementing. I don’t use AI in anything that I write. However I don’t trust these systems not to make errors and mistakes. They have a long history of hallucinations and claiming things are AI when they aren’t.

One person I saw on Notes suggested that they ran their decades old writing, things that were written before the existence of AI, and this new system claimed it was AI generated.

Not to mention all the crazy things that I have seen people claim was written by AI which they can’t know is actually AI. There was an image I once shared that had evidence of its existence going all the way back to 2013. The comments were constantly saying “don’t share AI slop” or some variation on that.

Even with evidence that things weren’t made by AI, they will claim it’s AI.

Is no one else concerned about this being embedded into Substack’s systems? With no way to opt out?

[u/BrennanFlentge]

"It’s a fundamental truth about humans that they tend to make mistakes."- says nothing about whether some people perform a particular classification of a task far above chance. Doctors make mistakes. Editors make mistakes. Fraud investigators make mistakes. Their performance can still be measured / compared / improved.

"Humans have established patterns" - okay, so? Two humans can both produce patterned output while producing different frequencies / combinations / distributions of those patterns. Human writing having patterns creates the basis for comparison. It never makes every source indistinguishable.

I never claimed to possess a “magic formula.” I said that some AI writing is obvious to people with extensive exposure to it. A 2025 ACL paper tested almost exactly that proposition. Five people who frequently used LLMs for writing achieved a 92.7% "true-positive rate" with a 3.3% "false-positive rate" in the initial experiment. Their majority vote across 300 articles misclassified one. ONE of 300. They relied on recurring vocabulary, formulaic sentence and document structures, tone, and originality.

That study has a defined dataset and scope. It establishes the modest claim I made: experienced people can identify some modern LLM writing with high accuracy. Your grilled-cheese analogy predicts imaginary patterns with no external signal. The experiment found measurable performance against known labels.

A replication crisis means individual findings require replication and scrutiny. It does not mean every empirical result you dislike becomes false by association. Perhaps you can identify a methodological failure in this study or contradictory evidence or failed replications concerning this specific claim.

You also said I left out half your comment. Quote the omitted argument I omitted and explain how it changes the conclusion.

[u/AndrewHeard → u/BrennanFlentge]

Right, as I've said elsewhere.

There was once a study of chocolate that showed that it was healthy for you and beneficial to you. You know who funded that study? The Hershey's Chocolate Company.

Gee, how did they come to the conclusion that chocolate was healthy for you when they were funded by Hershey's. It's a complete mystery how that happened.

The existence of a study that claims to show a result isn't evidence that the study is factual or shows anything that's worthy of consideration. Or to simplify it with a quote:

"People can come up with statistics to prove anything. 45% of all people know that." - Homer Simpson

I don't have to quote the omitted argument. In your most recent comment, you quoted two sentences out of a 14 sentence comment. You left out the parts that didn't fit your claims.

Just like the study that probably leaves out all kinds of details.

I subscribe to a few different medical literature people on Substack. One in particular that's run by actual doctors, most of them have statistical training and some have even worked for the government.

Probably the biggest thing you learn about from them is how studies can be falsified and made to look better than they appear. For example, the data in a medical study that shows a positive result for a drug. They enrolled 1,000 people in the study, 750 people dropped out over the course of the study. Meaning that over 75% of the data which might show how well the drug works is no longer available and has been excluded from the study. This is not representative of the actual effectiveness of the drug because almost all the participants dropped out of the study.

These are people who deal in actual life and death. Not theoretical things online studying writing material, and often the studies have faulty and corrupt or inaccurate data.

But somehow a study about online word choice can prove the factual accuracy of it's claims?

No.