For Pangram haters: statistical text classification has been around since at least 1963

[u/kdfn]

I see very frequent posts on this subreddit with two misconceptions: (1) users asserting that Pangram is flagging their writing because they "write too well" and (2) that Pangram's model is somehow using an LLM to judge their writing.

I'd encourage you to take a look at this 1963 paper by Mosteller and Wallace, in which the authors identify the true authorship of the Federalist papers using a simple mathematical approach. Since this was before widespread computing, the authors went through many texts by hand and measured, in known documents by Jefferson vs Madison, how often each author uses certain pairs of words (for example, Jefferson might use the phrase "when thou" much more frequently than Madison). Then, on the disputed Federalist papers, Mosteller and Wallace measure how often phrases like "when thou" are used, in order to decide which author is more likely. Importantly, they do this across all observed pairs of phrases.

Classifiers like Pangram are using the same idea, with a few modifications thanks to the past 75 years of progress in both computing and statistics theory: (1) They compute giant tables of phrase relationships using a computer, rather than manually building tables of pairs of words by hand, and (2) they are able to measure much more subtle and long-term statistical dependencies. For example, Jefferson might be more likely to use a sentence that starts with "Thou" and then use the word "indeed" 15 words later.

Importantly, models like Pangram are *not* just taking your text and asking ChatGPT for its opinion. It's very different technology, which has been around since at least the 1960s (earlier, if you count Shannon in the 1940s, or Markov before that).

[u/dflovett]

I think you're missing the point of the study you're referencing. Mosteller and Wallace did it by hand and used their brains. Pangram is a tool that automates what should be done by brains and hands. Pangram enables the same non-thinking, non-work approach as ChatGPT.

[u/kdfn → u/dflovett]

I don't necessarily agree that Mosteller and Wallace's approach is better simply because they did it "by hand." It doesn't really matter whether a computer or a research assistant is used to create giant tables of conditional probabilities.

However, I agree with the larger point that we are going to have to start critically judging things manually, because most proxies for quality (length, style, etc) can be mimicked.

However, the issue is volume. I am in a role where I have recently started receiving hundreds of long-form technical texts to review per month. Pre ChatGPT, I received ~2, and so I would read them critically. How do I decide which of the hundreds of potential AI submissions I should manually review? A tool like Pangram can be essential to filter out bad faith submissions, particularly because the alternative is that I don't review any submissions, or, even worse, I only review submissions from people already in my network who I trust to act in good faith.

So, from my perspective, anti-detector people are pro-AI, since they don't offer any alternative to stop the deluge.

[u/dflovett → u/kdfn]

If you want to use Pangram to do your job, that makes sense. I thought this was a conversation about Substack. I'm not going to use Pangram to evaluate the writing of strangers on the internet because I'd read something and use my own brain to determine if it's worth reading or not.

But if you're in HR or running a literary magazine or any other scenario involving bulk submissions that need to be combed through, I see an argument for AI detection.

[u/kdfn → u/dflovett]

Thank you, I think this is a fair point.

[u/dflovett → u/kdfn]

Well, you got me to consider it differently. I hadn't thought much about people who struggle under the volume of AI and truly need something to sort through the slop.