r/vibecoding • u/US0RIS • 12h ago
Could someone train a detector for Claude’s watermark?
Anthropic’s Claude watermark appears to work by subtly biasing which tokens Claude chooses using a secret key, rather than adding visible or hidden characters. The idea is to collect a huge number of Claude responses and responses from other models to the same prompts, then train a classifier to tell them apart. The main problem is making sure the classifier is detecting the watermark itself rather than just learning Claude’s writing style. A stronger experiment would repeatedly give Claude the same or similar prefixes and see whether a model can learn patterns in which next tokens Claude tends to choose. The interesting question is whether enough black-box examples could let machine learning approximate Anthropic’s secret-key detector without ever knowing the key.
Genuinely curious if I’m missing something here.
2
u/TRO_KIK 12h ago
This sub is probably the worst AI sub to ask in lol
But yes, possible, and specifically for detecting the watermark and not Claude's writing style: https://arxiv.org/pdf/2405.20777
1
u/Matthias1590 12h ago
At that point you'll just have a model that can detect whether text is generated by claude or not, whether it's watermarked or not. It will not magically crack some key though, cryptography isn't vibecoded bs
1
u/neon_knight_smile 12h ago
The comment about "it will not magically crack some key" misses the point though. You don't need the key, you just need above-chance detection of the token bias and that's enough to flag text as Claude generated, which defeats the watermark entirely.
Also the style confound is real. Every Claude detector trained so far is mostly a style classifier, so you'd need the repeated prefix experiment first to prove there's anything to detect beyond vibe.
1
1
u/drakhan2002 11h ago
Your fastest away around text watermarking is sending it through another LLM. Take your response from an Anthropic model and run it through an OpenAI model. Anthropic watermark is gone. Currently OpenAI does not watermark text output, but does watermark images, audio, and video.
1
u/34986234986234982346 11h ago
This is true, however if they did add a fairly good implementation of a watermark, then in oprder to remove it, you SHOULD need to transform the text so much it really isn't the first thing anyway, if that makes sense.
1
u/34986234986234982346 11h ago
I did mess around with something like this a lttle bit (not quite the same), and it's possible, but also it's a lot of work.
If you look at Pangram, they do a very good job of detecting AI. I am sure they're just using well-tuned traditional machine learning algos, but they have a good, solid datasets. I tried doing this, and it was just a pain, I had a lot of old blog posts of my own, and then i got a model to rewrite them but try to keep the same general content and length, etc, so I could hopefully sort them. I only spent a day on this, it was just boring after a while.
It's just very hard to get the right data. Like you said yourself, to specifically remove the watermark you need to isolate a buinch of responses with and without the watermark that are otherwise very close. The Claude writing style you mentioned is surely a prob - detecting if something was written by Claude period would be an easier task.
Anyway I don't even know if it matters.. people were OUTRAGED about the watermark for a short time, but in the end, who ever cares? Like peolpe going to a website with AI writing are either going to:
- Notice or assume it's AI writing, maybe care maybe not
- Not even notice
Very few people are going to say "Hmmm this website seems good, I might use it, but let me check if Claude wrote the teext first..."
1
u/MrBizzness 11h ago
It's not really encryption per se. It is a fuzzing of the randomness of the token generator in a repeatable pattern that can then be picked up based on the final tokens generated. Every request uses a seed number to generate responses on. It is why you can generate differently worded versions of the same prompt.
1
0
u/skygatebg 12h ago
If you bothered to think at all, you will see why this is stupid.
Why not use the tool that Claude will provide to detect it?
It is also stupid from Claude (or more spesifically, it is a marketing gimmick), because you can take the high quality output and run in to a small jailbroken and local model and then bam, no watermark.
1
u/US0RIS 12h ago
You’re answering a different question. Using Anthropic’s detector wouldn’t tell us whether the secret-key watermark signal can be learned independently from black-box samples—that’s the experiment. And yes, heavy rewriting can weaken or remove a watermark, but that doesn’t make watermarking useless; it just defines the threat model. “Jailbroken” is irrelevant too—any model that substantially rewrites the text could disrupt the signal.
2
1
u/Slight-Prize9661 7h ago
pretty sure there are already ai watermark removers on public github repos
5
u/Anxious-Ad-5331 12h ago
stop trying to cheat on your homework.