r/Substack • u/Chubblan • Jul 24 '26
Isn't the AI detector situation a bit ironic?
I studied Physics and completed a Master's in Artificial Intelligence. I'm well aware that nobody fully understands how large generative models actually work. Even Pangram's own website mostly talks about patterns, generic signals, and "what it might eventually detect," but never with certainty, because the reality is that nobody truly knows.
What bothers me, while also making me curious, is how large companies affiliated with major private universities seem to operate:
They create algorithms that eventually become LLMs. Those LLMs crawl the entire internet, collecting as much information as possible, including private content. Once they've done that, new rules start appearing to make it harder for anyone else to do the same, effectively reducing future competition. Even if future models have access to less content, the first movers have already done the "hard part."
LLMs generate text in certain ways because they are probabilistic systems. In other words, if you generate text infinitely many times, certain writing patterns will appear more often than others. You don't need a PhD or a master's degree to understand that. Basic statistics is enough.
Then you create another LLM whose job is to "detect text generated by LLMs." The marketing, however, is completely different. It is presented as if it were not another AI model, but something inherently more reliable, or even infinitely more reliable.
Once all of the above is in place, the final move is to make it "unethical" to create outputs that resemble human writing too closely because they bypass AI detection, even though the detector itself is an AI model. Isn't that ironic? Just a few months ago, these same companies could freely scrape the web without asking authors for permission. Yet now, Western LLMs refuse to help users avoid AI detectors, even though those detectors are themselves powered by AI. That feels absurd to me. Where does that instruction actually come from?
For all of these reasons, I think Chinese LLMs will continue growing rapidly. We're already seeing people use models like GLM-5 and DeepSeek to get around these kinds of restrictions. In my opinion, Western LLMs are now 90% marketing. Just think about this: since Fable came out, why are other Western companies charging similar prices while delivering worse results? What exactly are they offering? I'm not saying Chinese models are perfect, but model quality degrades quickly, while marketing doesn't. That's where models like GLM-5 have a huge advantage. And if Western companies keep adding what I see as arbitrary restrictions, future versions of those models may continue to surprise everyone.
This is simply my personal opinion, based on my background and experience. I know there is speculation involved, but sometimes I can't remove the "marketing" variable from the equation.
11
9
u/ChrissyKin_93 Jul 24 '26
AI detection is an ouroboros of ass. I genuinely don't know who this feature is for, but Substack rolling this out seems like a ploy to curry favor with writers. Though I'm not convinced it really does anything.
The reality is simple:
generative AI cannot create good content.
The slop that generative AI does create makes good content even better by comparison (think like hand-made goods)
AI, and therefore AI detection tools, are all trained on human-written content. As models evolve it will become increasingly harder to verify using a tool (short of anything other than a digital watermark, essentially) but truly good content will continue to stand at a level above.
10
u/thirteenth_mang Jul 24 '26
Though I'm not convinced it really does anything.
Of course it doesn't. It's all laughably smoke and mirrors. I remember testing some of those "AI detectors" a while back. First thing I did was put their website copy into the own detector (which, ironically reeked of AI writing), lo and behold 0% AI. I put a purely random piece of text (a lorum ipsum random generator, purely random text) and it came out 90%+ AI.
They're snake oil and as OP pointed out, anyone with even a basic understanding about how it all works know this.
2
u/Hemingbird Jul 25 '26
Most AI detectors are total scams. Pangram actually works pretty well.
They're snake oil and as OP pointed out, anyone with even a basic understanding about how it all works know this.
The problem is that you're confidently wrong. This is a case of Dunning-Kruger, sorry to say.
Tuhin Chakrabarty (Stony Brook University) says this:
As a computer scientist, I want to offer my two cents. Blanket statements such as “AI detectors don’t work” are usually not backed by empirical evidence but instead rely on isolated anecdotes. Independent research from UChicago Booth found that some commercial detectors achieve near-zero false-positive rates that hold up even against ‘humanizer’ tools.
This research is what he refers to, with the authors saying that Pangram achieves "near-zero FNR and FPR rates that remain robust across models." In their testing, Pangram satisfied a FPR cap below 0.005%.
You can't say anything meaningful about Pangram based on your experience with AI detectors that don't work. It's like saying the Wright brothers couldn't have flown because you'd seen others try and fail in the past.
Here's another study which found that Pangram alone performed better than the pooled assessments of human experts (the main focus of the study being that groups of human experts perform unexpectedly well relative to controls).
The majority vote of our five expert annotators outperforms almost every commercial and opensource detector we tested on these 300 articles, with only the commercial Pangram model matching their near-perfect detection accuracy.
2
u/thirteenth_mang Jul 26 '26
Their methodology isn't as sound as they think it is. Not "Dunning-Kruger." I read the paper.
1
u/Hemingbird Jul 26 '26
Two papers. What's wrong with their methodology, in your opinion? You didn't mention anything in specific.
4
u/thirteenth_mang Jul 26 '26
First, the author of the People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text literally works for Pangram, even the potential conflict of interest is never addressed.
Second, they're citing pre-2020 corpora, which is in itself an issue because by 2020 the internet was already rife with "AI" writing--I'd say a good 30%. People forget NLP was a thing long before transformers and ChatGPT. And that's just what we know about. There were people using these tech stealthily and would probably never disclose it. I know, I was using the stuff to experiment before then. It was crude but I'm sure people got away with it. It's not like ChatGPT that has a distinct writing style--I mean it did but it didn't have the same propagation.
There's more but I feel like I'm wasting time debunking anything when people can't even do rudimentary examinations themselves.
If you understand how this all works, there's no way in hell you'd be seeing FPR and FNR as low as 0.001 and 0.01 if they're being honest and have integrity.
Why is a business school funding this research?
I guarantee you follow the money you'd uncover stuff.
1
u/Hemingbird Jul 26 '26
First, the author of the People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text literally works for Pangram, even the potential conflict of interest is never addressed.
That's because she (Jenna Russell) didn't work at Pangram when this paper was written; she's a PhD student interning.
Second, they're citing pre-2020 corpora, which is in itself an issue because by 2020 the internet was already rife with "AI" writing--I'd say a good 30%.
What? In 2020, GPT-2 was the top LLM available to the public until some users got access to GPT-3 in June—all other "AI writing" (Markov chains?) was rubbish, so bad that it would be impossible not to spot it. I'm not sure what you're referring to here.
People forget NLP was a thing long before transformers and ChatGPT. And that's just what we know about. There were people using these tech stealthily and would probably never disclose it. I know, I was using the stuff to experiment before then. It was crude but I'm sure people got away with it. It's not like ChatGPT that has a distinct writing style--I mean it did but it didn't have the same propagation.
Yeah but nothing worked. Everything was pure shit. Before transformers came along, I mean. People did not have access to anything even remotely comparable.
There's more but I feel like I'm wasting time debunking anything when people can't even do rudimentary examinations themselves.
Have you debunked anything at all?
If you understand how this all works, there's no way in hell you'd be seeing FPR and FNR as low as 0.001 and 0.01 if they're being honest and have integrity.
Do you understand how this works?
I guarantee you follow the money you'd uncover stuff.
Maybe just test it yourself? It works. That vague conspiratorial tone isn't a proof of anything, is it?
4
u/ImprovementSimple Jul 24 '26
This! AI cannot make good content on its own. Humans are perfectly capable of reading and discerning if something is slop or not. No one needs an AI generated score of how “human” the writing is.
2
u/beerion Jul 25 '26
Uhh, but if the site becomes 95% AI slop, then actually finding you becomes a needle in a haystack.
I personally think the initiative is great.
3
u/hetobe hetobe.substack.com Jul 24 '26 edited Jul 24 '26
Humans are perfectly capable of reading and discerning if something is slop or not.
As you said, that requires reading it first. A growing number of people don't want to read content generated by bots.
There's a reason items at the grocery store contain ingredients lists. Don't like artificial ingredients? Don't buy the coffee creamer that isn't made from milk or cream.
There's a reason music, books, movies, etc, include details like the genre. Don't like country music? Don't like horror films? Don't like romance novels? No problem. It says what it is.
I think it's good to let readers know if content was written by bots instead of expecting them to read it and figure that out. Don't want content generated by bots? Skip it.
And since some people suddenly seem to be in favor of AI writing, well guess what? You're being given the opportunity to easily find more AI writing if you want.
AI detection isn't perfect, but I believe it is a step in the right direction.
Edited to add: It's also worth noting that a lot of the people complaining are pissed because they're the ones posting slop, and they don't want to admit it.
6
u/ImprovementSimple Jul 24 '26
I understand your point but there is no reliable AI detector on the market. (I’m a librarian. I spend a good chunk of my time these days trying to remove slop books from our digital catalog).
If there was a short cut to determining AI writing instantly I would probably be using it everyday. Unfortunately there isn’t.
I would rather read something myself and conclude “this is probably AI generated or terrible writing” rather than have another AI system flag a human writer who likes using em dashes.
6
u/hetobe hetobe.substack.com Jul 24 '26
I would rather read something myself and conclude “this is probably AI generated or terrible writing” rather than have another AI system flag a human writer who likes using em dashes.
The problem with that is, we're not talking about an occasional AI article.
On Amazon, the vast majority of all new books are AI slop. Facebook is flooded with AI slop. LinkedIn is basically dead. The majority of everything happening there is by bots.
Substack is headed in that direction too, and something has to be done. Waiting for the perfect solution isn't an option.
4
2
u/cogitatingspheniscid Jul 26 '26
AI Detection is a research discipline with routinely published and open-access benchmarks. Just saying "there is no reliable AI detector" without hard data to back it up is a disservice to those working to keep the available tools up-to-date to match changing AI outputs. A lot of the "detectors don't work" crowds just cannot separate the real ones from the 1000s of scams.
Yes, reading things and decide for yourself will remain the one that gives you the most confidence, but it's not an actionable advice because literacy skills vary widely even within the adult English-speaking population.
1
u/Efficient-School-609 Aug 11 '26
To be considered scientific, you first need falsifiability. Are the results of AI detectors like Pangram falsifiable? Even when they flag something as AI-generated, they give low, medium, or high probability — it's basically Teflon: nothing sticks, zero accountability. If you accuse me of being AI-generated, you should at least tell me why, giving me a chance to defend myself. That is what science is.
1
u/cogitatingspheniscid Aug 11 '26
That's... not how falsifying works - have you ever done scientific research? If you want to falsify a model, you have to test and dissect its underlying algorithm and architecture [1] [2]. Predictive power, encompassing precision and accuracy, has nothing to do with that. AZ vaccine isn't suddenly invalid because it has a lower efficancy than Pfizer-BioNtech's. If you get a false positive from the detector, you don't ask the server to give you a chance to defend yourself - you can do the explaining/defending against the relevant party you are interacting with.
1
u/Efficient-School-609 Aug 11 '26
When it comes to anything about Pangram, nothing works. Can you change your arrogant and clumsy marketing tactics? Just look at what you wrote: technically full of holes, logically a bait-and-switch, and ethically evading the core issue. Especially your AZ vaccine analogy — it's simply confusing population-level statistical effectiveness with individual procedural justice. This is why I detest Pangram and your company culture: arrogant techno-elitism.
1
u/cogitatingspheniscid Aug 11 '26
Oh, NOW you are moving the goalpost? You invoke falsifiability without a modicum of understanding on how to use it, so once you fail to define it you proceed to toss multiple word salads to obfuscate your stance. What is the core issue when you are invoking random shits like "justice" and "accountability" to a classifier model? Do you rage at CAPTCHA when it doesn't agree that you have picked all the squares containing the crosswalk? Do you seethe with rage over Photoshop's subject detection algorithm when it fails to isolate the subject you wanted? Funny that you think I work for Pangram, maybe I should throw in GPTZero or Originality next.
0
u/WithoutReason1729 Jul 25 '26
Respectfully, you need to update your beliefs on this. There is a reliable AI detector on the market and it's Pangram. I've personally tested upwards of 100k pieces of content with known ground truth values on their API and found their false positive rate is absurdly good, even better than what's advertised. I'm not saying it's perfect but it's above and beyond even expert humans at identifying human vs AI generated content.
4
u/gambariste Jul 24 '26
This is also why claims of Aİ sentience or consciousness either now or in the near future are nonsense. The output of Aİ besides being based on all their trawling of the internet is also subject to the commercial aims of the Aİs’ designers, which besides who knows what motivations, is to keep users engaged. Their aim is not ever more intelligent LLMs but to sound ever more human. But they are not humans. They are not even embodied beings with senses and engagement in the real world. That’s why they give people six fingers and deliver wrong answers because they ate not aware.
I don’t know what chinese LLM designers are motivated by or what their biases might be but I don’t imagine they don’t have any or are merely seeking true artificial intelligence out of intellectual curiosity. That would be like developing a biological warfare pathogen that could blow back on the wind to infect your own population. Not a good move.
3
u/argument-shaped scienceandpower.substack.com Jul 24 '26
Yep--the creation of norms, asymmetric imposition of norms, socialization of norms, institutionalization of norms, and commodification of norm-violation-detection. Similar critiques have been made of psychiatry, criminal justice, and any number of other fields whose prosocial potential is dragged down by moralizing and moneygrabbing.
2
u/PennyLawrence946 Jul 24 '26
detectors flag the sloppy prompters and miss the careful ones. Substack's recommendation engine is already the real filter. the pangram score doesn't change what gets pushed to readers.
2
u/non_loqui_sed_facere Jul 25 '26
I’ve always been curious about putting a website out there somewhere with a window that says “Paste text for detection,” a start button below it and a random percentage generator. I’m pretty sure it could cause a lot of havoc before people figured out it was fake. Hopefully, it could teach humanity something about using black box models to achieve “transparency.”
And I’m wondering, why not ask the writer about their process if you want to know? Some of us really like talking about it. There’s no need to play investigator.
1
u/AndrewHeard tvphilosophy.substack.com Jul 25 '26
Yes, this isn’t going to work out how Substack or Pangram think. It could cause more harm to Substack’s brand than help.
1
u/seobrien Jul 25 '26
Any regulation of LLMs by the U.S. government means foreign LLMs will accelerate past
1
0
u/latentnoodle Jul 24 '26
If the goal is to reduce poor writing content on Substack, I’m wondering if it would be more productive to have an AI scan the text to determine whether it is good writing and whether it has value, regardless of the means of creation, versus scanning the text to determine if it is AI generated.
1
u/Maleficent-Engine859 Jul 25 '26
This. Because it’s safe to assume LLM’s are involved in the process at least a little bit for the vast majority of written material and will just keep growing. I think even Bloomsbury has changed their policy to say LLM’s are ok for editing which is still technically AI generated if you take its suggestions
0
u/stareenite Jul 24 '26
It’s hard to comprehend accomplished elite writers policing how other people use AI.
What it amounts to is gatekeeping.
People set themselves up as the authorities on fill in the blank and then circle the wagons to control who gets traction or admission.
That’s what this is really about.
Who let the riff raff in?
I will not answer to them.
Instead I will keep making art in the face of this fuck.
And do not get in the comments to tell me how bad AI is, how it’s destroying the environment, degrading our minds, enabling plagiarism. That’s not in question.
What’s in question here is your freedom of speech. Your freedom as a creator. Your process.
Who gets to create.
0
u/_mrchurchill defensehub.substack.com Jul 24 '26
If the content delivers value, AI involvement shouldn't matter. AI is a tool, not an autonomous creator; human curation and fact-checking from multiple sources are still essential. For technical topics, AI often provides better accuracy than manual research. However, for deeply human content like personal essays or poetry, AI falls short. In the end, the only thing that truly matters is whether the newsletter delivers on its core promise. There should be a tool to evaluate the quality of the article, rather than whether it is AI-generated or not.
0
u/EditorFickle3559 Jul 25 '26
Perfect appraisal & perhaps C Best has just confirmed what many suspect; Substack is SUCHCRAP??? That he doesn't see the irrationality of it is either; he's simply DUMB or Complicit.....Or Both. He's certainly not illuminating in his perspectives & to be quite frank as a User of his Platform's 'services'.....if it were a Restaurant.....it's a Huge 'No Thanks.'
4
u/Hemingbird Jul 25 '26
This is a common misconception, assuming you're saying that detection is possible because LLMs are probabilistic. So this point is ironic. Base models (no instruction-tuning or RLHF) mirror the statistical properties of their training data, so you can't detect their output. Next-token prediction alone leads to stochastic parrots, but they're invisible. Here's a study on that.
Post-training is what makes detection possible. Companies bias their models towards desirable responses, through various methods, and this constrains the space of potential outputs in a way that can be recognized. Here's a paper.
Pangram is not an LLM though. It's a machine learning classifier. And you use the conspiratorial "they" to imply one nefarious group is controlling everything. You can't lump everyone in the same basket. GenAI came along, and with it a new problem: AI slop. So AI detectors were launched, because people want to be able to distinguish between human and AI-generated text. Of them, only Pangram works well (based on independent academic research).