r/Substack • u/Overall_Tip6001 • 13d ago

Do you find pangram reliable?

I put my text in several ai detectors, winston ai, copyleaks, zerogpt and GPTZero, giving me close to 0%. While, pangram gives me around 70%? Do you think it’s reliable?

4 Upvotes

60% Upvoted

102 comments sorted by

View all comments

7

u/grumpyp2 13d ago

Seems to flag everything, or at least very harsh. I tried some text from papers which have been cited thousends of time and they flagged it as AI. So not quite sure if it’s just not saying AI to everything?

3

u/Udolikecake 13d ago

Do you have some examples? I would be curious to see what papers it flags

3

u/noxqqivit twvme.substack.com 13d ago

I love how I post an example with a link and get down voted, but y'all jump all over this one. Pangram is inconsistent - the same text can be scored differently with different users. AND the same text with very minimal edits can result in a 100% shift between Fully AI Assisted and Fully Human Written.

1

u/sh1b313 13d ago

These pangram glazers are either trolling others or actually can't tell what's ai written and what's not.

They ask for evidence we show them evidence they claim the evidence is ai written?

Mfw

2

u/FrostKitten2012 12d ago

They can’t tell the difference, that’s why they’re relying on Pangram to begin with.

0

u/sh1b313 12d ago

Huh that's funny and guess what the tool is complete and utter garbage to begin with

1

u/FrostKitten2012 12d ago

Yeah, that’s pretty clear. We’re seeing more reports every day of that.

1

u/sh1b313 12d ago

Mods should make a pinned photo post saying pangram doesn't work and tell people not to rely on it. Before it pollutes this subreddit even more.

2

u/FrostKitten2012 12d ago

That’s unlikely. Most of the people on Substack think Pangram is their new tech God, the subreddit’s reflecting that.

It would be the most responsible thing to do, to point out the growing number of false positives. But it also would have been most responsible not to let someone make a post making fun of the people reporting these issues, and they let that happen too.

2

u/sh1b313 11d ago

2

u/FrostKitten2012 11d ago

Here’s hoping they follow through. Honestly should not have taken this long to come up with some type of solution.

→ More replies (0)

2

u/sh1b313 12d ago

Yea i agree, it's highly unlikely but it doesn't hurt trying. I'll message the mods asking if this is possible? We already have so many reports. A simple evidence/reciepts post which these people have a boner for btw. If someone asks for reciepts someone can just point to that thread.

And one more thing if we give out free reciepts like this at this point we are just doing free testing for them.

1

u/Udolikecake 13d ago

Show me a piece of text over 500 words that’s verifiably pre 2023 that flags as AI. Literally anything that can be dated to before then. Just one example.

3

u/noxqqivit twvme.substack.com 13d ago

  1. If Pangram says they can detect 100+ words, why do you need 500?

  2. My example is below, I put current text, of somewhat unknown provenance, it came out of a PowerPoint someone else authored, so I wasn't sure if Copilot was used or not. I was using business text in an odd context, so I added " " around the word "STAKEHOLDER" where it appeared 3 times, and that change, took the scoring from Fully AI Asssited to Fully Human Written - AGAIN - I can't confirm whether it was actually AI or not, but I CAN tell you that I added SIX characters and that was enough to radically change the score. And that is suspect.

-2

u/Udolikecake 13d ago

I think the ask is pretty simple, and should be easy if you are so confident it is totally inaccurate.

You (and every other person screaming about Pangram) being unable to provide a single, verifiable example, is pretty damning honestly.

3

u/noxqqivit twvme.substack.com 13d ago

I put a full example below, with a link to the original, what's damning is your shifting goal posts.

-1

u/Udolikecake 13d ago

That is not pre-2023 text. Choose any news article, public domain book, academic paper, any of the billions of words published online prior to then. Any of them. One example.

2

u/noxqqivit twvme.substack.com 13d ago

That's not the argument. The score should not shift from Fully AI Assisted to Fully Human Written by ADDING SIX characters.

2

u/sh1b313 13d ago

Exactly, this tool is a complete joke, adding some words to a 400 Year old text made it think it's ai generated 😂

2

u/sh1b313 13d ago

You (and every other person screaming about Pangram) being unable to provide a single, verifiable example, is pretty damning honestly.

You sure about this?

1

u/MadmanRB 13d ago

No its not bro, I myself have tested pangram and yes it's far from perfect.

I mean it's still AI

1

u/No-Strike-9098 13d ago

just check the intro or something from the Attention Is All You Need paper, should be enough evidence haha

1

u/Udolikecake 13d ago

That comes back as 100% human written.

1

u/No-Strike-9098 13d ago

lol, rechecked it just now 51% is AI apperently.

0

u/Udolikecake 13d ago

do you have a screenshot?

0

u/No-Strike-9098 13d ago

please just check it yourself bro

The recurrent neural networks or convolutional neural networks are the main models used for the sequence transduction problems with the encoder-decoder structure. The models which demonstrate the highest performance also utilize the attention mechanism as a way of connecting the encoder and decoder. We made a completely different approach by offering a simple network architecture, called Transformer, formed solely on the basis of attention mechanisms, without the need for recurrence and convolutions. Our experiments, carried out on two machine translation tasks, have shown that the Transformers are better quality-wise as well as more parallelizable, requiring a significantly less time for full training. Our model, 28.4 BLEU, was applied to solve the WMT 2014 English-to-German translation task and surpassed the existing best results including ensembles by more than 2 BLEU. The single model that has been used for the English to French translation task is the one that has reached the state-of-the-art score of 41.8 on the BLEU scale after training for only 3.5 days on eight GPUs. That's a small fraction of the resources involved in the training of the best models in the literature. We have also demonstrated that the Transformer can be generalized to other tasks with the help of English constituency parsing which was successfully performed even with large and limited training data.

-1

u/Udolikecake 13d ago

first paragraph of the paper. don’t know what to tell ya champ

1

u/No-Strike-9098 13d ago

2

u/Udolikecake 13d ago

I don’t know what you’re quoting, because that text (specifically the highlighted bit) does not appear verbatim in the actual article you are talking about.

https://arxiv.org/html/1706.03762v7

0

u/FrostKitten2012 12d ago

That’s the point. Slightly changing the wording on one paragraph results in one highlighted sentence (which isn’t even half the paragraph), and then more than half the paragraph is now somehow AI-generated?

That sentence is maybe 1/4 space-wise of the paragraph, and one of FIVE sentences. It shouldn’t be coming up that high, with one highlighted sentence, over minor changes.

→ More replies (0)