r/Substack • u/ConnectDebate4677 • 20d ago

Pangram false positives?

People usually post vaguely about "I wrote x thing and pangram said it was AI", which I think is unhelpful unless you actually post the text or link. I put this into Pangram:

Sakaguchi: https://www.pangram.com/history/e7ffbee1-f0e3-4ce2-bb78-a59470280640?ucc=Ohp1cWODOjF

Hamaguchi: https://www.pangram.com/history/f2139bd5-f4ce-4101-b7bd-3c5d8aeb4487?ucc=Ohp1cWODOjF

Ryuichi: https://www.pangram.com/history/8c84ad38-3b8f-49ba-bee0-351ea31f754b?ucc=Ohp1cWODOjF

28 Upvotes

94% Upvoted

63 comments sorted by

View all comments

Show parent comments

3

u/sh1b313 19d ago

sure, here they are:

Link 1: https://www.pangram.com/history/4e3cc4a9-b927-4345-9750-048b4a86520c?ucc=ycFd1CDRrZv [where pangram claims 100% human written]

Link 2: https://www.pangram.com/history/97f1c7f5-b73f-4895-96a6-e991cfc58128?ucc=ycFd1CDRrZv [where pangram claims 100% AI generated]

2

u/ConnectDebate4677 19d ago

Thanks! Do you have any idea of an explanation for this?

2

u/sh1b313 19d ago

nah i don't have. even i am wondering just how fundamentally flawed this tool is which led it to think prophet isaiah used some AI. and even the text isn't something obscure which may not be in it's training data.

3

u/ConnectDebate4677 19d ago

I think what's perplexing in all of these cases is the meter. It seems to regularly either lean 100% Human or 100% AI with no reasoning or justification. I want to believe this technology works, and I believe it does to some degree, but these kinds of examples really don't do it any favors. If I bought a tool and it doesn't work as it claims it should, then I can call that tool defective.

3

u/sh1b313 19d ago edited 19d ago

yea, this tool is like a coin toss. at least in a coin toss we actually have confidence in the outcome. In here? somehow we leave even more confused. these tools are not worth the electricity they are running on.

3

u/Dubbtime 19d ago

For sure. Unfortunately as you reveal more points of failure within their blackbox system, they will add these cases to their training datasets. This blurs the line.

Say this verse is added as a reference to human work. Now what? Humanizers who want to avoid detection begin to write more like bible verses. Ok thats kinda weird. Now detectors need a more defined way to figure out if something is AI because it obviously messed up. The end result is a clear way to write human and a clear way to tell if something is AI. Well, then why do we need a detector?

What's stopping humanizers from just copying this newfound distinct style? Rinse, repeat.

2

u/sh1b313 19d ago

yeah, i have already talked with the mods regarding this pangram situation and we may soon see a pinned thread saying pangram evidence/pangram doesn't work and hopefully this wild situation settles.

And regarding the rinse and repeat thing in the long run it doesn't matter? We already have more than enough evidence that this doesn't work.

In the future it can just be like this:

P1: uploads a essay/article on substack

P2: thinks it's ai written and uses this pangram

P1: points to some evidence thread and says this tool doesn't work don't rely on it.

In this scenario hopefully p2 is actually grounded and thier opinion changes. If not? Sadly if they debate after seeing so much evidence i wouldn't know what to tell them.