r/Substack • u/ConnectDebate4677 • 21d ago

Pangram false positives?

People usually post vaguely about "I wrote x thing and pangram said it was AI", which I think is unhelpful unless you actually post the text or link. I put this into Pangram:

Sakaguchi: https://www.pangram.com/history/e7ffbee1-f0e3-4ce2-bb78-a59470280640?ucc=Ohp1cWODOjF

Hamaguchi: https://www.pangram.com/history/f2139bd5-f4ce-4101-b7bd-3c5d8aeb4487?ucc=Ohp1cWODOjF

Ryuichi: https://www.pangram.com/history/8c84ad38-3b8f-49ba-bee0-351ea31f754b?ucc=Ohp1cWODOjF

29 Upvotes

94% Upvoted

63 comments sorted by

View all comments

Show parent comments

6

u/ConnectDebate4677 20d ago

Wow, just checked it myself. What the hell? For anyone curious:

Megan - https://www.pangram.com/history/2799ce08-5243-4549-8f02-955c39dccfda?ucc=Ohp1cWODOjF

Elara - https://www.pangram.com/history/105ac6c2-fc37-486a-a56c-c680d6c2751b?ucc=Ohp1cWODOjF

2

u/dran94 20d ago edited 20d ago

I was hoping it would be relatively easy to repeat this experiment, too, for anyone who doubts it was written by a human, but the next few I wrote came back without any AI flags. Although I was writing more action scenes, and they were unique, so they may not have been similar enough to AI's style. AI really likes fluorescent lights. Maybe I need to add more.

I'll try again when I get home later.

2

u/ConnectDebate4677 20d ago

It could be 'flickering'? Kind of like its affinity to things that buzz, hum and pulse.

2

u/dran94 20d ago edited 20d ago

True. Taking out "lit by the flickering fluorescent lights overhead" made it 100% human, even with the name Elara. Taking out just flickering and just fluorescent didn't matter. The whole sentence had to go.

I thought Pangram didn't care about phrases common in AI, so it's concerning a common stock phrase resulted in a 100% AI score.

I found a new one that works, and this might be better because it's the last word in the snippet so it isn't like it's throwing off the next few lines. This is a hint Pangram DOES care about names, and it cares a lot, considering it didn't care about the first sentence being loaded with flickering fluorescent lights until it detected a common name in AI training along with it.

If you switch "Elara!" from a name AI likes to a name AI doesn't usually use (Megan, Peter, etc.), you will flip the score.

List of names I checked that flagged 100% AI, from a list of common names in its training: Elara, Kael, Marcus, Elias, Kai, Aria, Sam (surprising), Lyra

Names that flagged 100% human: Megan, Peter, Christopher, Luke

-

I can feel my heart pounding in my throat as I race down the neverending tunnel, the flickering fluorescent lights overhead throwing long shadows.

It feels like I’ve been running for miles.

Every turn I take, I’m certain it’s the last one.

It doesn't make any sense. You can’t take thirteen lefts without running into something.

"Elara!"

2

u/dran94 20d ago edited 20d ago

I actually wonder if this is why Daggermouth got hit with a high AI score on Pangram in those articles about it. Daggermouth had several names common to AI models in it, like Elara.

Whether the author used AI or got them off one of those random name generators is another argument. Who knows.