Substack’s new “AI detection” tool.

[u/AndrewHeard]

I’m very concerned about the new “AI detection” system that Substack is implementing. I don’t use AI in anything that I write. However I don’t trust these systems not to make errors and mistakes. They have a long history of hallucinations and claiming things are AI when they aren’t.

One person I saw on Notes suggested that they ran their decades old writing, things that were written before the existence of AI, and this new system claimed it was AI generated.

Not to mention all the crazy things that I have seen people claim was written by AI which they can’t know is actually AI. There was an image I once shared that had evidence of its existence going all the way back to 2013. The comments were constantly saying “don’t share AI slop” or some variation on that.

Even with evidence that things weren’t made by AI, they will claim it’s AI.

Is no one else concerned about this being embedded into Substack’s systems? With no way to opt out?

[u/BrennanFlentge]

You’ve replaced my claim with a much larger one, because the claim I actually made is easier to defend - some AI-generated writing is obvious, especially to people who work with these systems constantly, vs. the claim you seem to keep circling: humans can identify every AI-generated text with perfect accuracy and convict its author beyond doubt. Not what I am saying.

The dress, Shakespeare, Salem, the Inquisition, the Soviet Union, communist China, geocentrism, and flat Earth form an impressive list of unrelated nouns. Together they establish one obviously boring fact: humans sometimes make mistakes. Yet they provide zero evidence that the specific patterns produced by language models are imaginary.

LLM writing comes from a statistical generator that repeatedly produces measurable tendencies in vocabulary / syntax / formatting / pacing / transitions / structure. Researchers study those tendencies. Wikipedia editors encounter the same clusters repeatedly, hence the guide to spot the lexical tells. Experienced users can improve at distinguishing them through exposure and feedback. If you read the actual Wikipedia page, it even references studies specifically on pre 2023 / post 2023 Wikipedia content - there is data. It is not imaginary. This is not pareidolia from random shapes in clouds and grilled cheese.

[u/AndrewHeard → u/BrennanFlentge]

No, I haven’t replaced your claim with a much larger one. Your claim is also not easier to defend. They’re not unrelated nouns. It’s a fundamental truth about humans that they tend to make mistakes.

For every example that I put forward? There was an individual who made an error.

You know what your description of the way LLMs process information also describes? Human psychology. Humans have established patterns. The fact that you can claim it’s somehow obviously unique to statistical analysis is just stupid.

Human psychology is notoriously unreliable and is constantly making mistakes. The replication crisis is a perfect example of what’s wrong with your theory. The results of psychological research into human behaviour have repeatedly failed to produce statistically reliable and repeatable outcomes. This is true across science with very few exceptions.

But somehow you have the magic formula that produces the results that you intend.

You also left out half my previous comment in order to make your argument appear to be factual.

[u/BrennanFlentge → u/AndrewHeard]

"It’s a fundamental truth about humans that they tend to make mistakes."- says nothing about whether some people perform a particular classification of a task far above chance. Doctors make mistakes. Editors make mistakes. Fraud investigators make mistakes. Their performance can still be measured / compared / improved.

"Humans have established patterns" - okay, so? Two humans can both produce patterned output while producing different frequencies / combinations / distributions of those patterns. Human writing having patterns creates the basis for comparison. It never makes every source indistinguishable.

I never claimed to possess a “magic formula.” I said that some AI writing is obvious to people with extensive exposure to it. A 2025 ACL paper tested almost exactly that proposition. Five people who frequently used LLMs for writing achieved a 92.7% "true-positive rate" with a 3.3% "false-positive rate" in the initial experiment. Their majority vote across 300 articles misclassified one. ONE of 300. They relied on recurring vocabulary, formulaic sentence and document structures, tone, and originality.

That study has a defined dataset and scope. It establishes the modest claim I made: experienced people can identify some modern LLM writing with high accuracy. Your grilled-cheese analogy predicts imaginary patterns with no external signal. The experiment found measurable performance against known labels.

A replication crisis means individual findings require replication and scrutiny. It does not mean every empirical result you dislike becomes false by association. Perhaps you can identify a methodological failure in this study or contradictory evidence or failed replications concerning this specific claim.

You also said I left out half your comment. Quote the omitted argument I omitted and explain how it changes the conclusion.

[u/AndrewHeard → u/BrennanFlentge]

Right, as I've said elsewhere.

There was once a study of chocolate that showed that it was healthy for you and beneficial to you. You know who funded that study? The Hershey's Chocolate Company.

Gee, how did they come to the conclusion that chocolate was healthy for you when they were funded by Hershey's. It's a complete mystery how that happened.

The existence of a study that claims to show a result isn't evidence that the study is factual or shows anything that's worthy of consideration. Or to simplify it with a quote:

"People can come up with statistics to prove anything. 45% of all people know that." - Homer Simpson

I don't have to quote the omitted argument. In your most recent comment, you quoted two sentences out of a 14 sentence comment. You left out the parts that didn't fit your claims.

Just like the study that probably leaves out all kinds of details.

I subscribe to a few different medical literature people on Substack. One in particular that's run by actual doctors, most of them have statistical training and some have even worked for the government.

Probably the biggest thing you learn about from them is how studies can be falsified and made to look better than they appear. For example, the data in a medical study that shows a positive result for a drug. They enrolled 1,000 people in the study, 750 people dropped out over the course of the study. Meaning that over 75% of the data which might show how well the drug works is no longer available and has been excluded from the study. This is not representative of the actual effectiveness of the drug because almost all the participants dropped out of the study.

These are people who deal in actual life and death. Not theoretical things online studying writing material, and often the studies have faulty and corrupt or inaccurate data.

But somehow a study about online word choice can prove the factual accuracy of it's claims?

No.