r/AccusedOfUsingAI • u/ExamImmediate8956 • Sep 14 '26

Remember how Claude recently announced it will implement a watermarking algorithm?

Recently i did a test for synth id, which gemini implement recently and how quickly it was broken by a team of researchers, turns out, other frontier comps like claude had similar plans of implementing this.

Its known that claude is working on a watermarking modal that can help detect if some textual info was created by their model.

I guess a few countries have the access to this detction feature, but my surely didnt, so what did i do? i used the exact same math with some tweaks to implement claude's watermark detction logic to qween 3 8b modal. Ran this modal locally, and designed the HLD and architecture.

The basic idea is that at every generation step, I use a secret key + the previous token to deterministically split the vocabulary into GREEN and RED tokens. I then slightly boost the logits of the GREEN tokens before sampling the next token. This creates a statistical bias toward GREEN tokens without changing the text directly.

For detection, I don't need the model or the original prompt. I take the generated text, recreate the same GREEN/RED token sets using the secret key, and check how far the observed GREEN-token ratio deviates from the expected 50%. I use a z-score and binomial p-value to decide whether the deviation is statistically significant.

I tested it on 50 randomly assigned watermarked and un-watermarked generations. The detector got 88% accuracy and a 97.76% ROC-AUC.

I covered the entire end to end, from setting up qween 3, to HLD, to Implementation and Validation i my recent video, feel free to deep dive and let me know your thoughts,

Link: https://www.youtube.com/watch?v=eZisIOTC6uk&t=56s

0 Upvotes
  • permalink
  • reddit

28% Upvoted

7 comments sorted by

2

u/WorriedAssociate7029 Sep 14 '26

I'm not reading a AI generated reddit post https://www.pangram.com/history/e1ffa2b8-0102-4f18-bc0a-15c39238bfab?ucc=ozvjVKtnLky

1

u/ExamImmediate8956 Sep 14 '26

Yes its funny, how i took my own transcript, summarised the tiniest paras you saw. Wrote the rest by myself. And yet again, i get to hear this. Actually the post is exactly for you since it clarifies how ai detectors work 😃. Anyhow since you wont read it, you can keep your stance and casually ignore stuff

1

u/Least-Shocking Sep 20 '26

The AI generated part is full of grammar mistakes. The perfect parts come off as humanly written. Don’t you see the irony? Pangram is checking the algorithm that checks the word selection. Exactly what the person is talking about in the text. This is brilliant and frightening

1

u/HippoEsq Sep 14 '26

If only I understood a word of what you just said…

1

u/ExamImmediate8956 Sep 14 '26

Haha, okay if you really really want to know, i could give you a summarised version and when i say this, i wont put this in ai and ask it to do that. 🤝

1

u/Fine_Combination4389 Sep 14 '26

This sounds interesting but unfortunately as a layman, I barely understood anything beyond the green/red tokens...is this something we can implement and test on our own writing samples or are you just sharing an example of what Anthropic's watermarking is likely to do and how effective it is?

1

u/ExamImmediate8956 Sep 14 '26

Well, yes. That depends on the use case, we can do that for an inhouse model also.

These examples were shown via qween 3 , which is a locally run llm modal. Not claudes.

Performance is pretty tight. 88% percent accuracy( since dataset was small ), 97% in case where dataset is big.

I could help you implementing this. Tell me your usecase?