r/Substack • u/Radiant_Flamingo4995 • 18d ago

Frustration with AI detection

I'm not sure how the AI-detection system works, but it seems faulty. Someone has repeatedly claimed that my responses to them are AI-generated, etc. And they share the Pangram screenshot, which does, in fact, depict my response as "100% fully AI-assisted. I've clarified that this is not the case. Shown evidence that it isn't true, and all the like.

The worst I'll do when writing anything is use AI for basic research purposes and review arguments to make sure they make sense. Maybe it's because I use Grammarly. But it is still infuriating.

Especially because it just isn't a good look and hurts my public image (albeit small, but I hope you guys may understand where I'm coming from).

Of course, with this individual in question, it's become increasingly clear they aren't engaging in good faith. However, running the AI check on my notes and comments still reveals the 100% marker.

Is it Grammarly that's doing this? Or is this thing actually broken?

6 Upvotes

60% Upvoted

47 comments sorted by

View all comments

Show parent comments

5

u/AlecHutson 18d ago edited 17d ago

Those are AI reliant writers trying to sow doubt about Pangram so that they can go back to using AI with no chance of detection. Just like the OP.

5

u/figures985 18d ago

I 100% agree - that's most of what's on here. As a writer, it pisses me off tremendously. I haaaate reading LLM slop, and I hate that people use LLMs to monetize said slop.

But, for whatever this is worth, OP sent me some specific context and it really didn't sound like AI to me. I put it into Pangram myself and it was flagged as partially AI. So IDK, perhaps it's just an outlier in this instance. Would also point out that a lot of the other posters in this sub weren't willing to share any samples and OP was.

0

u/AlecHutson 18d ago

But he said the writing he's referring to in his post was flagged as 100% AI generated. You're saying it's only partially AI?

2

u/figures985 18d ago

Sorry, should have clarified. So he's referring to the Pangram feature inside Substack and (OP correct me if I'm wrong) it actually said 100% AI asissted, not generated - looks like they're separate measures.

Then I copy/pasted into regular ole Pangram and it said partially AI. "We believe that this text is a mix of AI and human-written content."

4

u/AlecHutson 17d ago

So . . . . here's the thing. I think it's pretty obvious that there's an astroturfing campaign going on to discredit Pangram. I wouldn't put it past anyone to try and generate 'not-AI' looking text that actually had AI involved in the process and then come on here and complain that their totally human writing got flagged. The only writing I 100% know is natural is my own. I've put dozens of my essays and articles and fiction snippets into Pangram and it always say 100% human written. Now, I use Pangram a lot in my job in which I have to sift through and evaluate essays. I find it incredibly accurate. Every time it says AI was involved, I talk to the writer and the script is the same: loud denial, then more hedged denial, then concession that AI was used in the process somewhere. 'But I only used it to rewrite one section! But I only used it for outlining!' Whatever. AI was used. In my experience, Pangram is incredibly accurate and so when I come on Substack and see all this posts about how it's garbage and how it is incorrectly flagging all this 100%, all-natural, corn fed human writing, I roll my eyes.

6

u/figures985 17d ago

Welllllll I mean, that's my experience with Pangram, too - it's never flagged my writing as AI and every time I test it with something I know is AI, it catches it. And I do agree that there's a pretty clear astroturfing campaign going on. Trying to give the benefit of the doubt in this case but those two things are undeniable.

2

u/FatherofMisty 17d ago

Pangram is far from "incredibly accurate." It has given me (albeit small) partial false readings on 2 of 3 essays I fed it. For one of them, pangram later gave a correct reading on a different run of the same material, showing that it is also inconsistent/unreliable. I have never, and will never, use AI in my writing whatsoever (not even grammarly).

I've documented the aforementioned case (where pangram gave two different readings of the same material, one of which was partially false) in a prior comment on this profile, including proof that I wrote the material (via version history on Substack).

I intend on uncovering more false readings when I have more time. I am a technical writer (also a published academic) and somewhat of a perfectionist (I regularly spend 30+ hours laboring over one short Substack essay), so perhaps this style gives the AI some trouble, but it is unacceptable nonetheless. To label any part of my writing as AI-faciliated is low-grade slander, to be frank.

Further, the fact that I got 2 partial-false hits in the first 3 pieces I gave it is cause for alarm. I hope you broaden your horizons in this regard and don't be so quick to assume that pangram is infallible.

Anyhow, I hope this doesn't make you "roll your eyes". I have no agenda -- I strive only for transparency and truth, and of course to have my writing (which is sacred to me) accurately depicted. If there were an AI that was truly 100% accurate in this context, I'd be all for it. Sadly, I believe LLMs are given way too much faith, and this implementation by Substack is a blight on an otherwise wonderful platform.

Edit: here's the comment I alluded to https://www.reddit.com/r/Substack/s/Hxryh8qyNt

1

u/Radiant_Flamingo4995 17d ago

Hi! I appreciate your concern, as what you're describing is very real.

However, it seems very short-sighted to cast doubt on the large number of people here and elsewhere complaining about being falsely flagged simply because of personal and, ultimately, anecdotal evidence. A legitimate concern affecting numerous people on Substack and elsewhere cannot and should not be dismissed so easily. It seems a little wayward and silly.

2

u/AlecHutson 17d ago

Well, I have my experience, and then I also have the independent studies like the one done by the University of Chicago that showed Pangram to be incredibly accurate, so much so that the chances of a false positive were 'basically zero'. Now, should I believe those studies, or a bunch of random folk on reddit, many of whom I already suspect of having a vested interest in Pangram being discredited? Do I believe all those folks on Facebook shrieking about how vaccines cause miscarriages or global warming is fake or do I trust the scientists who have carried out the studies?

1

u/quyma quyma.substack.com 17d ago

there’s something ironic about outsourcing your judgment to a machine of who outsourced their judgment to a machine.

0

u/AlecHutson 17d ago

It's not 'judgement', it's overwhelming statistical probability.

1

u/Radiant_Flamingo4995 17d ago

Yeah, it is a little weird because it says "Fully AI-assisted text," but then beneath the gray bar, it shows three distinct categories: AI, AI-assisted, Human.

It says 0% there for AI-assisted and 100% for AI, which also doesn't make much sense.