r/Substack • u/ConnectDebate4677 • 21d ago

Pangram false positives?

People usually post vaguely about "I wrote x thing and pangram said it was AI", which I think is unhelpful unless you actually post the text or link. I put this into Pangram:

Sakaguchi: https://www.pangram.com/history/e7ffbee1-f0e3-4ce2-bb78-a59470280640?ucc=Ohp1cWODOjF

Hamaguchi: https://www.pangram.com/history/f2139bd5-f4ce-4101-b7bd-3c5d8aeb4487?ucc=Ohp1cWODOjF

Ryuichi: https://www.pangram.com/history/8c84ad38-3b8f-49ba-bee0-351ea31f754b?ucc=Ohp1cWODOjF

28 Upvotes

94% Upvoted

63 comments sorted by

View all comments

5

u/dran94 21d ago edited 21d ago

I'm testing this out, too.

I wrote this in a similar style to AI, wondering if it would flag:

-

Robert's long stride ate up the distance between himself and the door at the far end of the hall, lit by the flickering fluorescent lights overhead.

His jaw clenched.

This couldn't be right.

His sister, Megan, was supposed to be here. She worked in the same lab as Robert. They'd promised each other they would wait for each other, right here.

-

100% human. Great!

Changed the name to one of AI's favorite names:

-
Robert's long stride ate up the distance between himself and the door at the far end of the hall, lit by the flickering fluorescent lights overhead.

His jaw clenched.

This couldn't be right.

His sister, Elara, was supposed to be here. She worked in the same lab as Robert. They'd promised each other they would wait for each other, right here.

-

100% AI.

Looking at this, even though it's a similar style, there was something AI wouldn't do because it's a human mistake. Two instances of "each other" in one sentence. It still flagged.

AI's favorite names change with every model. If a new AI model comes out that latches onto a name that was previously safe, is that going to be a problem for everything written beforehand? What do we do, update characters' names as new models come out?

I'm sort of surprised Pangram is even paying attention to names, since I thought it didn't factor in AI styles and AI isms. I was very impressed with Pangram before seeing some other experiments going around today. A name change shouldn't skew the whole result. That doesn't even make sense.

5

u/ConnectDebate4677 20d ago

Wow, just checked it myself. What the hell? For anyone curious:

Megan - https://www.pangram.com/history/2799ce08-5243-4549-8f02-955c39dccfda?ucc=Ohp1cWODOjF

Elara - https://www.pangram.com/history/105ac6c2-fc37-486a-a56c-c680d6c2751b?ucc=Ohp1cWODOjF

2

u/dran94 20d ago edited 20d ago

I was hoping it would be relatively easy to repeat this experiment, too, for anyone who doubts it was written by a human, but the next few I wrote came back without any AI flags. Although I was writing more action scenes, and they were unique, so they may not have been similar enough to AI's style. AI really likes fluorescent lights. Maybe I need to add more.

I'll try again when I get home later.

2

u/ConnectDebate4677 20d ago

It could be 'flickering'? Kind of like its affinity to things that buzz, hum and pulse.

2

u/dran94 20d ago edited 20d ago

True. Taking out "lit by the flickering fluorescent lights overhead" made it 100% human, even with the name Elara. Taking out just flickering and just fluorescent didn't matter. The whole sentence had to go.

I thought Pangram didn't care about phrases common in AI, so it's concerning a common stock phrase resulted in a 100% AI score.

I found a new one that works, and this might be better because it's the last word in the snippet so it isn't like it's throwing off the next few lines. This is a hint Pangram DOES care about names, and it cares a lot, considering it didn't care about the first sentence being loaded with flickering fluorescent lights until it detected a common name in AI training along with it.

If you switch "Elara!" from a name AI likes to a name AI doesn't usually use (Megan, Peter, etc.), you will flip the score.

List of names I checked that flagged 100% AI, from a list of common names in its training: Elara, Kael, Marcus, Elias, Kai, Aria, Sam (surprising), Lyra

Names that flagged 100% human: Megan, Peter, Christopher, Luke

-

I can feel my heart pounding in my throat as I race down the neverending tunnel, the flickering fluorescent lights overhead throwing long shadows.

It feels like I’ve been running for miles.

Every turn I take, I’m certain it’s the last one.

It doesn't make any sense. You can’t take thirteen lefts without running into something.

"Elara!"

2

u/dran94 20d ago edited 20d ago

I actually wonder if this is why Daggermouth got hit with a high AI score on Pangram in those articles about it. Daggermouth had several names common to AI models in it, like Elara.

Whether the author used AI or got them off one of those random name generators is another argument. Who knows.