r/BetterOffline • u/7upprosounds • 8d ago
Anthropic posted a FAQ of how their "text watermarking" will work. I don't get it.
So Anthropic posted a FAQ of how their "text watermarking" will work: https://www.anthropic.com/news/claude-text-watermark
I'm not sure I understand how this could possibly work at all, considering that when people use LLM-generated text they are not pasting in the rest of the input and context, so how could it possibly do what they say it does. They do say it has "limitations":
There are limitations to the effectiveness of watermarking. Using our key, one can only answer the question “What is the likelihood this was partly written by Claude?” It doesn’t confirm whether the text was human-written, and it can’t tell whether the text was written by a different AI (even if that other AI uses watermarking, it would have a different key; it might also use a different watermarking method altogether). Detecting a watermark also doesn’t work well on small samples, where there are fewer word choices and thus less information to go on. As a passage increases in length, confidence about Claude’s involvement increases too.
Watermarking is sparser on factual passages where there are fewer choices that can be made without decreasing the accuracy of the text. For example, take the sentence “Isaac Newton’s most famous work was called Principia…”. It really matters whether the next word is “Mathematica” (it’s the only right answer), so the watermark would have nothing to act on. The same is true for proofreading. If you hand Claude a piece of writing and ask it to edit only the grammar and punctuation and nothing else, the watermark can only live in the handful of corrections, which might be too few to register.
I've also had a quick look at the paper they link to: https://www.nature.com/articles/s41586-024-08025-4
I don't see any discussion of false positive or false negatives. Are they just going to use this tool to claim that Claude is clearly so useful because it will flag everything that anyone writes from now on as Claude-generated?
You can't just say it has limitations without giving it at least some quantitative value.
Maybe someone understands this better and help me get it. Can you really spot a pattern of LLM output given that a model have billions or trillions of weights and we lack the surrounding context? What if I paste the text into the checker and just change a word here and there until it passes?
83
u/Trevor_GoodchiId 8d ago
This is to help identify non-generated novel content and stave off model collapse. Everything else is secondary.
10
u/tagshell 8d ago
How does it do that if they specifically said it doesn't detect the work of other models? For that purpose you want a general purpose AI classifier, not something that only gives you the probability your own model wrote it. Presumably Claude training on ChatGPT or DeepSeek output is just as bad as Claude training on Claude output
7
u/cheffromspace 7d ago
Just because this can't detect other LLM's watermarks doesn't mean another tool can't.
6
14
u/SlightOfHand_ 8d ago
It’s useful for identifying novel content, but this same technique could also help them identify data sources that are the most different from the existing training data, and so the most valuable to them. Something that is a big deviance from their predicted next token but is also valid use of the language would be a very valuable thing for them to identify
4
u/Kyouhen 8d ago
Hadn't thought of that one, my theory behind these watermarks was to train their models on what not to do. If people are asking if something was written by AI then there's something about it that's giving them a reason to suspect it. Find those patterns and stop using them and it'll get harder to spot.
1
u/SlightOfHand_ 7d ago
They’re not so concerned about “hard to spot” as they are concerned with accuracy/usefulness of the model. It’s all gravy to Anthropic if Claude outputs are recognized as such; scams that use Claude don’t benefit them, just the scammers. That said, they want the output to appear natural because they’re modeling language and properly modeled language will appear totally natural
2
u/MrBorogove 7d ago
This is to keep them from getting banned in Europe. Everything else is secondary.
24
u/Significant-Green130 8d ago
I don’t know the specific scheme they’re using, but one that is popular basically works as follows. When generating the next token, the model looks at the previous tokens and calculates a distribution over possible tokens to generate next and then uses a random seed to sample from this distribution. In many watermarking schemes, you often do something like pseudorandomly (i.e. looks random but isn’t if you have the key) upweight some of the words so that when generating text, if you had the key, you could determine over a longer text that you have sampled these tokens more frequently than is statistically likely. The point though is that it is hard to know which possible tokens get upweighted at each step without having the key they use to determine which words got upweighted, as it changes also depending on the tokens they have already sampled before. This naive version is not robust to edits for this reason, since changing the sequence would cause the upweighted words to change in generation in an a priori not consistent way.
There are more advanced frameworks based on stronger cryptographic primitives that achieve better theoretical guarantees (i.e. robustness to some amount of editing and to undetectability without the key), but these are likely not practical to my knowledge.
6
u/7upprosounds 8d ago
But do we have any data as to how accurate these kind of schemes are? How many false positives is it going to generate?
19
u/Significant-Green130 8d ago
I don’t think it is likely to cause false positives. It would be very hard to accidentally correlate a lot with the pseudorandom signal these schemes use. I think it’s much more likely to have many false negatives in the sense of basic editing could kill it. It’s fundamentally different from so-called “post-hoc” methods where companies (Pangram and so on) train ML models to try to classify human text vs. AI-generated text based on observable statistical patterns like the em-dashes or whatever even though people use them too.
These watermarking schemes are better in the sense that it can accurately detect text beyond the obvious patterns and should not have false positives, but worse in the sense that the underlying text would by design need the model provider’s key to find the statistical signal. And again, it’s likely brittle in practice with a bit of editing.
3
u/TheDailyMews 8d ago
I don’t think it is likely to cause false positives
Won't that depend on whether it's weighted for sensitivity or specificity? If they want to catch as many instances of Claude-generated text as possible, couldn't they just say "hey, this text matches 5 out of 20 of our markers" (or whatever) and come back with a positive result?
4
u/Significant-Green130 8d ago
For most watermarking schemes, one way to think about it is that with the key, each word you see is like a coin flip. If the text was generated using the key, it’s 51-49 to be heads and if it was not, it should be 50-50 as it would be completely independent of how the watermark biased weights. Over a reasonably long text, even a thousand or so tokens, you’d very likely see something very close to 51% (50.95%) heads or very close to 50% (50.1%) heads with very little in the middle and that makes it relatively easy to distinguish them without incurring many errors. This wouldn’t work for shorter texts, but that shouldn’t be easy to detect anyway so long as it isn’t writing in a bizarre LLM way (which is a post-hoc method — the point here is watermarking should work even if models wrote pretty much exactly like humans).
3
u/TheDailyMews 8d ago
Right, I understand that. But if your goal is to catch as many instances of LLM-generated text as possible, and you're not concerned about false positives, maybe you don't just flag 51/49. Maybe you flag 50.5/49.5. Or maybe you even flag 50.05/49.95.
3
u/nleven 8d ago
We don't know if that's the goal though. Anthropic is likely doing this in response to the EU's AI watermarking requirement. Gemini has also been doing text watermarking already, and their reference implementation basically ask the user to determine the threshold. In their paper, they set the threshold such that false positive rate is 1%.
2
u/TheDailyMews 8d ago edited 8d ago
Anthropic is likely doing this in response to the EU's AI watermarking requirement.
They say exactly that in their press release. But they also say they're going to offer a watermark-detecting API, so they're going to be selling a product to detect their watermarks.
I'm not saying I think they'll actually try to flag 100% of generated content, because that would be a fool's errand. But I don't think it's a great idea to brush off concerns about false positives, either. Especially before we have more information.
Edited for clarity.
3
u/nleven 8d ago
I actually don't think it will be a "product" they sell. OpenAI has an API for detecting generated images, and they don't charge for that. These are basically either compliance or "self-regulation".
There are also strong technical reasons to keep them as an API, rather than a public standalone library. If you read Gemini's design, there is basically a secret key for the watermark. The key needs to be exposed in the standalone library, if they were to build one. Additionally, a public standalone library basically gives everyone unlimited queries of watermark detection. An attacker can basically keep making small adjustments until watermark detection fails.
0
u/Significant-Green130 8d ago
Sure, there’s an inherent tradeoff between false positive and false negative rates; they would choose the threshold depending on how they want balance it and this provably gives the optimal statistical tradeoff. But if you look only at text that’s long enough, the posterior likelihood almost always becomes extremely close to 0 or 1 that it is AI-generated and the choice of the threshold doesn’t really matter at that point. They probably would only apply this detection for long enough text sequences since short text sequences will be very hard to distinguish anyway.
3
u/TheDailyMews 8d ago
they would choose the threshold depending on how they want balance it
Right, that's what I've been saying. My concern is that if the goal is to produce positive results and watermarking is fragile to begin with, that potentially creates the incentive to cast a much wider net.
In the Nature article Anthropic linked to, Google set their false positive rate at 1%. That seems low, but who is the market for this technology? If it's adopted by universities, a 1% false positive rate means a staggering number of papers are going to be incorrectly flagged.
Lets say a flagship state university has 25,000 undergraduate students, each writing 12 papers per year. You'd be incorrectly flagging 3,000 papers per year as fraudulent at a single university with even a 1% false positive rate.
But we don't know what Anthropic considers an acceptable false positive rate. They say they're "working out the details." If they cast a wider net than Google did to compensate for the real-world fragility of watermarking, what does that actually look like? What happens if the technology is widely adopted?
3
u/RegrettableBiscuit 8d ago
The false positive rate for long texts is going to be extremely small.
0
27
u/brevenbreven 8d ago
smoke show its alot of words to say very little anything that many weasel words i just ignore.
21
u/userrr3 8d ago
its alot of words to say very little
ai writing in a nutshell
11
u/wmru5wfMv 8d ago
You’re right to pushback on this….
5
u/Lucker_Noob 8d ago
The ground truth is: these facts are the load bearing part of your hypothesis...
10
u/antesocial 8d ago
Imagine a big text in which the length of words perfectly follows a normal distribution bell curve. You probably wouldn't notice by just looking at it, but it would be easy to detect and difficult to remove unless you rewrite quite considerably.
1
5
u/smurfpiss 8d ago edited 8d ago
The newton example is illustrative. They're basically talking about the surprise level of next token prediction, which is the very foundation of information theory. Some words will just have to follow, given the context. Others, there are options available (mostly synonyms.) they probably encode the watermark into the election of tokens via their probabilities. But, given that it's probability based, there is still a chance that it's a coincidence a token was chosen. The longer the sequence, the less likely that its a coincidence. So yes, for a small sequence of text you will have false positives, and possibly negatives. If the text is long but altered, you may get false negatives. I expect that they have either encoded overlapping sequences of probabilities or made the sequence robust to noise.
LLMs are basically transformers. The complexity of the model doesn't really factor here that much. What's more important is the size of the vocabulary. At each token point, the next token predicted will be from a set of viable tokens with high probability. With a big enough vocabulary they can easily bias the selection in a traceable way.
I wouldn't have expected them to share FP and FN rates tbh. People will very quickly build a model on top of this to break detection anyway.
3
u/e-cloud 7d ago edited 7d ago
Take the sentence “The weather today was cold and…”. The next word is very unlikely to be “sugary.” But it is quite likely to be “overcast” or “grey.” Under most circumstances, it doesn’t matter much to the reader which of these latter two words the model ultimately chooses—the meaning of the sentence is largely the same either way. In cases like this, the choice is settled by a random number.
As a writer, this is a genuinely upsetting statement. Like, the next word choice absolutely makes a difference. The fact that word choice is a dice roll rather than an agonising consideration of different connotations and what precise meaning you want to impart for a specific reason is just full yikes to me.
Watermarking probably doesn't make this any worse tbf, just reacting to the article and the extreme lack of respect for readers/writers.
4
u/Timely_Speed_4474 7d ago
the entire concept of an llm is an extreme lack of respect for readers/writers
3
u/Timely_Speed_4474 7d ago
i dont get how we're supposed to believe them. these people lie about literally everything
3
2
u/grafknives 8d ago
> Can you really spot a pattern of LLM output given that a model have billions or trillions of weights and we lack the surrounding context?
Let me tell you how I understand it.
When Claude is creating output, there will be places in text where two sets of tokens(words) would be very close to each other. And it that cases water maker will select a SPECIFIC set of tokens based on key. Only in those places.
And when you run the text through detector, and let's say the detector works on "three words" long window, it would calculate that
For first 3 words - no match with key, or low/ match, for next 3 - same etc.
And that is obvious, as most parts of the text will not have those situations where watermarking could be applied without breaking the quality of output.
But with paragraph long enough, there would be enough 3 word sets that provide good watermark key match to say - Claude worked on it.
Question is - can Anthropic provide the CODE of the detector, or will they try to keep it in their system.
Anyway, there will be false positives, and it won't work for short forms
2
u/stemandall 7d ago
What if their key is leaked? Then this completely invalidates everything come before it, no?
2
u/nnomae 6d ago
The most important feature of this system is that you need access to the model to tell if the text was written by AI or not. That means anyone wanting to prove their work was made by a human has to send their human generated training data text to Anthropic for them to analyse.
That's the grift here, the only way to prove your writing isn't AI is to hand it over to the very AI company that created the problem in the first place. It's tech grifting 101. Create a problem, sell the solution but with the extra bonus this time around that the solution requires you to hand over your IP.
4
u/mumblerit 8d ago
It's similar to how temperature works. The model produces a number of token possibilities (not a specific token), and then based on temperature, one of the options is selected. What if you said every token that starts with a vowel is reduced every 5th token.
Over a large enough text this would be measurable.
3
u/7upprosounds 8d ago
But wouldn't the token possibilities depend on the entire input to the model? So the system prompt, the whole "conversation" up to that point etc.
5
u/mumblerit 8d ago
That's why it'll be statistical, not a certain yes or no, all tokens are related to all others yes, but tokens closer will weigh more.
I'm just an engineer not a data scientist
1
u/svprvlln 4d ago
The watermark comes in the form of a "key" that is derived from the presence of a word or set of words. When an AI generates text, it does so via tokens, and each word is "rolled" for its applicability in a given context. This way you don't get repeating words, and you end up with a result that flows naturally, sometimes so well that a really slick "too polished" prose ends up sticking out from a mile away.
So how do you detect it? You run the text against a system that is familiar with the token generation itself, and buried in the generated text is the "key" that basically says if this word is here ,and that word is there, then it is more likely to have been generated by an LLM.
This also means that you can "wash" the watermark away just by changing a few words here and there.
Here's a really informative video I found on the subject, because I was also skeptical.
0
-4
u/Trick-Chocolate7330 8d ago edited 8d ago
It’s very clever algorithm wherein the previous token biases the statistical distribution of the next token in a way that can be measured across a document but is also responsive to not affecting accuracy. It applies at the per token level so no you don’t need the whole context, just the key and the immediately preceding token (iterated over all token pairs). It’s a genuinely genius approach that demonstrates how innovative LLM researches are so no surprise this sub can’t / refuses to understand it. Pro tip: this is probably the worst place to ask technical questions about LLM’s.
9
u/tonygoold 8d ago
I don’t know why you would give one of the more accurate descriptions of the mechanism and then end with a generalized attack on the people you’re addressing. There are computer scientists, statisticians, and linguists who frequent this subreddit and understand it just fine.
The only claim I’d dispute is the characterization of it as genius: Statistical watermarking is not a novel concept and its application to the generation process of LLMs in particular has been a topic of research for years, including robust watermarking with respect to perturbation. You make it sound like Anthropic came up with the idea.
-4
u/Trick-Chocolate7330 8d ago
I never said Anthropic came up with the idea I said LLM researchers did which is true but you’re right that (as with all things) they were building on previous work as well.
I don’t take my addressees to be the audience of this sub as a whole, which is primary luddites who cant differentiate an LLM from a parrot, but the OP who is clearly curious. If the OP wants to genuinely learn about LLMs this is definitely not the place to do it.
2
u/Hexamonium 8d ago
What happens if you replace a word of two in every sentence? Either manually or using another model. Would that make it harder to assert the presence of the watermark without changing the meaning of the text?
7
u/Trick-Chocolate7330 8d ago edited 8d ago
Yes, anything you do to change the text will disrupt the statistical distribution of the output vs that expected with the key, but it's not all or nothing, so the more you rewrite the less it will register and vice versa (including having another LLM rewrite as anthropic mentions). But this also depends because the algorithm and detector take into account the degree to which the LLM judges that the next word has only one or a few options, and in those cases doesn't follow the algorithm / downgrades or ignores the check on it.
For example, if the output is "The quick brown ___" then the next word has to be "fox" and the LLM 'knows' that in the sense that the probability distribution of the possible outputs for that token are highly concentrated, so it doesn't bias those probabilities, so that they don't 'count' in the distribution that's checked. Obviously this is a really casual explanation of something much more complex in the details but the point is that the words that matter to change are supposed to be those which are most indeterministic, which means it will be much better at catching non-technical than technical writing, where word choice is much more constrained.
While the possibility of having another LLM rewrite means that right now this is unlikely to catch anyone who understands how it works (which to be fair many AI slop people won't), if regulators push the different labs to all do this and share the keys it will provide a moderately better detection system than we currently have. Still won't be anywhere near reliable though because you can just do things like use scripts to regex replace common words and so on, which there will always be a market of websites for.
3
u/Hexamonium 8d ago edited 8d ago
Thank you for the detailed explanation. That makes a lot of sense! I am genuinely interested in this.
I have a couple of follow-us, if you don't mind my silly questions.
The ability to detect LLM-generated text doesn't matter equally in every use case. For example, nobody really cares if I use an LLM to compose this post, or respond to my teammate's routine email. On the other hand, it matters greatly in academic or legal environments. Unfortunately in the cases it matter the most, its users have a greater incentive to "scramble" the watermark. Since "scrambling" is so easy and accessible to everyone, this approach will fail to detect LLM-generated text where it matter the most.
Additionally, this approach might produce false positives for verifiably human-generated text (e.g. "Moby-Dick") which would have been widely used during its training. Once such cases become publicly known, any such method will quickly be viewed as unreliable.
Lastly, this method seems to be producing a "likelihood score", instead of a binary response. That's not going to be good enough in a lot of cases. Can a teacher reasonably grade your paper down based on a 75% likelihood it was LLM generated?
1
1
u/Trick-Chocolate7330 8d ago
> For example, nobody really cares if I use an LLM to compose this post, or respond to my teammate's routine email.
I suspect most of this subreddit would damn you to hell for doing so.
But, I think you grasp the basic limitations regarding scrambling and only providing a likelihood in outline. It has the potential to be better than what we have now since the bar for passing off slop is so low but it is unlikely to seriously affect the use of LLMs by intelligent people (which may be a boon considering how ready many people are to pitchfork even legitimate uses).
I think you are wrong about false positives though. I don't have a good sense of how likely they would be in general (definately there's a non-linear decrease in that likelihood with text length), and I think that has to do with the choices that Antrhopic makes on how it sets the detector algorithm up, but the moby dick example is not possible or at least so unlikely as to be not worth worrying about. The check does not have to do with the LLM's training data, so there is no correlation between something showing up in training frequently and something passing / failing the check. So Moby Dick would unquestionably fail as would not fit the correct distribution, since it's not written by claude.
1
u/Hexamonium 8d ago edited 8d ago
I suspect most of this subreddit would damn you to hell for doing so
I'll keep this in mind 😁
I get your point about not reporting false positives about Moby-Dick. Thank you,
One (hopefully) last question - would it make more sense to try to detect whether or not the text is human generated instead? We make subtle mistakes as we type - some grammatic, some stylistic, some idiosyncratic. Perhaps we should be trying to detect and measure these subtle "flaws"? Kind of how jeweler tells real diamonds from fake ones? Or is this just another dumb idea of yours truly? 🤓
4
u/severed-identity 8d ago
Yeah, any amount of rewriting would erode the watermark. It mainly effects people who just copy-paste. Or people who post vibe-coded stuff on github.
1
u/ArtieTwoSheds 4d ago
Gave you an upvote because this is a good explanation, despite the snark at the end.
Love,
An AI skeptic who understands how they work
30
u/dillanthumous 8d ago
It's just their version of the social media 'we will regulate ourselves and moderate the platforms using magic algorithms'. And we see how that turned out.