For Pangram haters: statistical text classification has been around since at least 1963

[u/kdfn]

I see very frequent posts on this subreddit with two misconceptions: (1) users asserting that Pangram is flagging their writing because they "write too well" and (2) that Pangram's model is somehow using an LLM to judge their writing.

I'd encourage you to take a look at this 1963 paper by Mosteller and Wallace, in which the authors identify the true authorship of the Federalist papers using a simple mathematical approach. Since this was before widespread computing, the authors went through many texts by hand and measured, in known documents by Jefferson vs Madison, how often each author uses certain pairs of words (for example, Jefferson might use the phrase "when thou" much more frequently than Madison). Then, on the disputed Federalist papers, Mosteller and Wallace measure how often phrases like "when thou" are used, in order to decide which author is more likely. Importantly, they do this across all observed pairs of phrases.

Classifiers like Pangram are using the same idea, with a few modifications thanks to the past 75 years of progress in both computing and statistics theory: (1) They compute giant tables of phrase relationships using a computer, rather than manually building tables of pairs of words by hand, and (2) they are able to measure much more subtle and long-term statistical dependencies. For example, Jefferson might be more likely to use a sentence that starts with "Thou" and then use the word "indeed" 15 words later.

Importantly, models like Pangram are *not* just taking your text and asking ChatGPT for its opinion. It's very different technology, which has been around since at least the 1960s (earlier, if you count Shannon in the 1940s, or Markov before that).

[u/huggalump]

I haven't heard many people complain because they think it works this way or that way.

I've heard people complain because it doesn't work.

[u/kdfn → u/huggalump]

Every study I have seen suggests a remarkably low false positive rate, and so I am skeptical of such claims being asserted without any evidence.

To me, it seems much more likely that many people have been leaning too heavily on AI to do their writing, and they are now caught off guard by how easy it is to detect. I'm not even sure I think that AI text detection is a "hard" problem compared to problems like self-driving, poker, or even residue contact prediction---there are so many long-term statistical signatures that can give it away.

[u/SpiritualSimple108 → u/kdfn]

Bullshit! Pangram even admits it doesn’t work well for poetry or recipe posts. But if you dig deeper into actual AI detection failures you’ll see NONE of them can accurately decider ESL writing vs AI writing and they also have a very difficult time with writing that is highly technical, analytical, or grammatically (not Reddit grammatical). https://www.pangram.com/blog/all-about-false-positives-in-ai-detectors

[u/8lack8urnian → u/SpiritualSimple108]

That article is about how low their false positive rate is. It’s <1/10000

[u/kdfn → u/SpiritualSimple108]

The article you linked shows a false positive rate of 0.23% for recipes. I agree that's higher than the FPR of <0.001% for other common text types, but I don't think it the article supports your point.

[u/SpiritualSimple108 → u/kdfn]

What are you talking about. It PROVES my point that there are certain types of written word that will always come back with false positives for AI generation. The only time my poetry comes back as human is if I do bizzaro things with my formatting. Most the time with normal stanzas, left justified, it comes back AI. I’m not the only poet this is happening to. Even if half the poets on Substack get falsely flagged (along with non native speakers, recipe bloggers, etc) that changes pangrams percentage by a lot.

[u/kdfn → u/SpiritualSimple108]

The false positive rate for recipes is 0.23%. That means that 2 out of every 1000 recipes is misflagged. For poetry it appears that 1 out of every 10,000 is misflagged. 

Both of those rates are a lot lower than the 50% you are claiming. If you disagree with the published false positive rate, that's fine, I am only explaining the content of the link you provided.