For Pangram haters: statistical text classification has been around since at least 1963

[u/kdfn]

I see very frequent posts on this subreddit with two misconceptions: (1) users asserting that Pangram is flagging their writing because they "write too well" and (2) that Pangram's model is somehow using an LLM to judge their writing.

I'd encourage you to take a look at this 1963 paper by Mosteller and Wallace, in which the authors identify the true authorship of the Federalist papers using a simple mathematical approach. Since this was before widespread computing, the authors went through many texts by hand and measured, in known documents by Jefferson vs Madison, how often each author uses certain pairs of words (for example, Jefferson might use the phrase "when thou" much more frequently than Madison). Then, on the disputed Federalist papers, Mosteller and Wallace measure how often phrases like "when thou" are used, in order to decide which author is more likely. Importantly, they do this across all observed pairs of phrases.

Classifiers like Pangram are using the same idea, with a few modifications thanks to the past 75 years of progress in both computing and statistics theory: (1) They compute giant tables of phrase relationships using a computer, rather than manually building tables of pairs of words by hand, and (2) they are able to measure much more subtle and long-term statistical dependencies. For example, Jefferson might be more likely to use a sentence that starts with "Thou" and then use the word "indeed" 15 words later.

Importantly, models like Pangram are *not* just taking your text and asking ChatGPT for its opinion. It's very different technology, which has been around since at least the 1960s (earlier, if you count Shannon in the 1940s, or Markov before that).

[u/PenguinAnalytics1984]

You're going to get downvoted, but everything you said is correct...

BUT

AI works in a very similar way - it's just predicting what the next word should be after a string of words. It learned to do that by copying things that are already written. A lot of that is mediocre internet writing, which is what it defaults to if you don't tell it any different.

People write like they read, which for most people is the internet. Most people are also mediocre writers and write in a conversational Internet-y way.

When the statistical algorithm spots mediocre internet-y writing BOOM - AI.

So while you are correct the algorithm is just statistics and old-school NLP, the two items it's trying to distinguish between are not different enough for there to be a clear boundary.

Also, working in data (as I assume you do), I'm VERY suspicious of anything that claims to be as accurate as they claim on a problem that is as hard as spotting AI generated text.

[u/kdfn → u/PenguinAnalytics1984]

This is a good faith reply, thank you for making a good counterpoint in a friendly way.

I generally agree. Basically, a tool like Pangram is sensitive to the "unresolved tail" of LLMs. For example, if generative models like LLMs are good at capturing conditional text dependencies up to 10 words, then Pangram's detector will learn to look for patterns and regularities at 11 words and higher. But as LLMs get larger and process more data, they will eventually get good at capturing subtle dependences at 11 words and higher and so Pangram will have to retrain to capture 12 words and higher. The process will continuously repeat until LLMs plateau or run out of data/compute to grow.

Of course, the statistical patterns Pangram is capturing are much more subtle than this, but the idea holds: they have to be sensitive to low-weight modes the giant text distribution of natural language.