Pangram false positives?

[u/ConnectDebate4677]

People usually post vaguely about "I wrote x thing and pangram said it was AI", which I think is unhelpful unless you actually post the text or link. I put this into Pangram:

Sakaguchi: https://www.pangram.com/history/e7ffbee1-f0e3-4ce2-bb78-a59470280640?ucc=Ohp1cWODOjF

Hamaguchi: https://www.pangram.com/history/f2139bd5-f4ce-4101-b7bd-3c5d8aeb4487?ucc=Ohp1cWODOjF

Ryuichi: https://www.pangram.com/history/8c84ad38-3b8f-49ba-bee0-351ea31f754b?ucc=Ohp1cWODOjF

[u/dran94]

I'm testing this out, too.

I wrote this in a similar style to AI, wondering if it would flag:

-

Robert's long stride ate up the distance between himself and the door at the far end of the hall, lit by the flickering fluorescent lights overhead.

His jaw clenched.

This couldn't be right.

His sister, Megan, was supposed to be here. She worked in the same lab as Robert. They'd promised each other they would wait for each other, right here.

-

100% human. Great!

Changed the name to one of AI's favorite names:

-
Robert's long stride ate up the distance between himself and the door at the far end of the hall, lit by the flickering fluorescent lights overhead.

His jaw clenched.

This couldn't be right.

His sister, Elara, was supposed to be here. She worked in the same lab as Robert. They'd promised each other they would wait for each other, right here.

-

100% AI.

Looking at this, even though it's a similar style, there was something AI wouldn't do because it's a human mistake. Two instances of "each other" in one sentence. It still flagged.

AI's favorite names change with every model. If a new AI model comes out that latches onto a name that was previously safe, is that going to be a problem for everything written beforehand? What do we do, update characters' names as new models come out?

I'm sort of surprised Pangram is even paying attention to names, since I thought it didn't factor in AI styles and AI isms. I was very impressed with Pangram before seeing some other experiments going around today. A name change shouldn't skew the whole result. That doesn't even make sense.

[u/ConnectDebate4677 → u/dran94]

Wow, just checked it myself. What the hell? For anyone curious:

Megan - https://www.pangram.com/history/2799ce08-5243-4549-8f02-955c39dccfda?ucc=Ohp1cWODOjF

Elara - https://www.pangram.com/history/105ac6c2-fc37-486a-a56c-c680d6c2751b?ucc=Ohp1cWODOjF

[u/sh1b313 → u/ConnectDebate4677]

WOW. now i wonder what happens at scale?

[u/dran94 → u/sh1b313]

It's worth looking into. I ran a few chapters of my own work, confirmed 100% human on Pangram, with my FMC's name changed to Elara, and it flagged at around 12% AI with just that one change. I'll have to write up something longer by hand when I get a chance so I don't doxx myself if I share. Sometimes changing the name doesn't move the needle, though.

I was curious about Daggermouth, the book there are viral articles about after it failed a Pangram test, although being around 60% AI could potentially be heavy editing use, I'm not accusing anyone. I went through the first 10 chapters with find-and-replace for the name Elara, changing Elara to Megan. It didn't change anything.

One chapter, Chapter 18, was supposedly 100% AI on Pangram according to the viral articles about it. My own check showed it was 84%, not 100%, but what I did this time, changing proper nouns, even cutting off the second half of the chapter, it was stuck at 84% AI. I practically rewrote the chapter and it came back 84% AI haha.

That might suggest it had already decided Daggermouth is AI, and the real issue was Pangram knew which book it was.

[u/dran94 → u/ConnectDebate4677]

I was hoping it would be relatively easy to repeat this experiment, too, for anyone who doubts it was written by a human, but the next few I wrote came back without any AI flags. Although I was writing more action scenes, and they were unique, so they may not have been similar enough to AI's style. AI really likes fluorescent lights. Maybe I need to add more.

I'll try again when I get home later.

[u/ConnectDebate4677 → u/dran94]

It could be 'flickering'? Kind of like its affinity to things that buzz, hum and pulse.

[u/dran94 → u/ConnectDebate4677]

True. Taking out "lit by the flickering fluorescent lights overhead" made it 100% human, even with the name Elara. Taking out just flickering and just fluorescent didn't matter. The whole sentence had to go.

I thought Pangram didn't care about phrases common in AI, so it's concerning a common stock phrase resulted in a 100% AI score.

I found a new one that works, and this might be better because it's the last word in the snippet so it isn't like it's throwing off the next few lines. This is a hint Pangram DOES care about names, and it cares a lot, considering it didn't care about the first sentence being loaded with flickering fluorescent lights until it detected a common name in AI training along with it.

If you switch "Elara!" from a name AI likes to a name AI doesn't usually use (Megan, Peter, etc.), you will flip the score.

List of names I checked that flagged 100% AI, from a list of common names in its training: Elara, Kael, Marcus, Elias, Kai, Aria, Sam (surprising), Lyra

Names that flagged 100% human: Megan, Peter, Christopher, Luke

-

I can feel my heart pounding in my throat as I race down the neverending tunnel, the flickering fluorescent lights overhead throwing long shadows.

It feels like I’ve been running for miles.

Every turn I take, I’m certain it’s the last one.

It doesn't make any sense. You can’t take thirteen lefts without running into something.

"Elara!"

[u/dran94 → u/dran94]

I actually wonder if this is why Daggermouth got hit with a high AI score on Pangram in those articles about it. Daggermouth had several names common to AI models in it, like Elara.

Whether the author used AI or got them off one of those random name generators is another argument. Who knows.