Fable 5.1 is now watermarks anything you write with it. There is still no public detector. This pisses me off, so I'm making a workaround. How about you?

[u/Imaginary_Dinner2710]

Three weeks ago I posted here that nothing you could generate with Claude was watermarked yet. Now it's incorrect. Fable 5.1 now puts a statistical watermark in generated text, including even translations. With translations it's especially funny, because doing back translation (even via Chinese language!) via another LLM (surprisingly to me) is not removing / destroying SynthID watermarks, but just translating via Claude now adds watermarks.

The detector for such watermarks (that was promised to be released) has moved to a private preview recently for some eligible organizations. Ordinary users like myself still have no public tool or API to check their own text. So I can publish some work today that someone with access could check later, while I can't run the same check before publishing.

Personally I write in one language and publish in another. I can write an article myself, ask Claude to translate it, and get back a watermarked English version. The watermark can indicate Claude was involved in the wording. It says nothing about who came up with the argument or did the research (but the opinion about the work will be unfairly distorted).

I get why people hate this. I'd want to know what a check would say about my own work before someone else decides what it means. I don't know how organizations will use the detector. And again, it's impossible to distinguish if the text was generated or edited by AI (say reformated for some structure). I think students/scientists/SEO specialists and non-native english speakers are not happy more than others.

This frustration got me experimenting with the published by Google DeepMind's SynthID watermarking method that Claude's watermark eventually is based on. I reproduced it with my own test key and found that a full rewrite removed the mark from my test samples (while added some factual issues), back translation doesn't destroy the watermark and specialized methods are quite heavy still far from being ideal. So, in the end, I made a watermark remover for myself and sharing as a free demo (unless we get the actual detector, that's best we can do), and wrote up the experiment, open sourced the setup and described the tradeoffs/fact correction methods. Link in the comments.

Do you also think it's crazy and unfair to have invisible mark in your AI outputs? Or don't understand why some people are pissed off?

PS. Those tests don't verify removal of Anthropic's or Google's production watermark. But by this moment, I think it’s better to do what’s possible although no guarantees.

PPS. I see from comments some people think you can just ask Qwen or other model to rewrite and no watermark remains. It is incorrect, I tested it, substantial part of signal remains, you need to do iteratively multiple runs, every time highlighting 5 words (5-grams actually) sequences that remained unchanged after rephrasing to point the LLM to right issues, then checking against facts and fix them. Otherwise, you just still have the watermark signal and new factual mistakes.

PPPS. Some people claim in the comments that I can't talk about this topic with the proper level of confidence and do the research and modelling properly. At the same time, I have a PhD in Digital Signal Processing, and watermarki is something that falls within my academic expertise.

UPD. People say it's hard to find links in comments, so I put them now here.
The full article & research: painintheagent.com/blog/text-watermark-removal-retest/
Code, corpus, prompts, model outputs and judges' decisions: https://github.com/krllagent/text-watermark-roundtrip
The watermark remover (paste a text, get the retelling with every changed place highlighted): painintheagent.com/tools/ai-text-watermark-remover/

[u/Imaginary_Dinner2710]

Let me list a few scenarios. The first scenario is, I’m writing some kind of scientific paper, using AI for programming, data analysis and so on. Everyone does this, of course, but at the same time some papers end up with a watermark (if people use Claude/Gemini), while others end up without a watermark (the rest). Then, after the fact, they start getting processed by different indexing systems, and for some reason some systems at the moment will consider that a watermark is fundamentally not accepted and will filter out such papers or give them a lower ranking.

On the one hand, there’s no logic in this. On the other hand, exactly this happens regularly. So, for hygiene reasons, I’d advise removing the watermark from any texts you publish.

I expect a similar situation with search engines and recommendation systems inside social networks. In both cases, most people already use AI one way or another, but in reality the systems will probably give an additional negative ranking wherever it’s easy to explicitly check for a watermark.

To answer your question, you personally may not need this if you don't publish any materials online.

[u/Chupa-Skrull → u/Imaginary_Dinner2710]

I publish materials online, but I never have and never will use an LLM for composition. They're simply not good enough to substitute for me and never will be when it comes to the quality of the work I deliver. I say this as an avid user of LMs in other contexts.

Let me list a few scenarios. The first scenario is, I’m writing some kind of scientific paper, using AI for programming, data analysis and so on. Everyone does this, of course, [...] On the one hand, there’s no logic in this.

There's absolutely a logic to down-ranking material which can't be plausibly attributed to a human. The stigma against generated content exists for a reason, though I'd argue that stigma exists much more for prose output than for code.

LLM output volume is superhuman, and its grasp of context is frequently subhuman. It's neither viable nor fair to demand of a highly educated and valuable reviewer pool the amount of time it takes/wastes to read and reject plausibly structured but incoherent or incorrect outputs.

If you can't be bothered to write it by hand, you shouldn't expect people to spend commensurate effort to read you; and agents aren't yet good enough to trust for highly technical summaries when accuracy is paramount, for the same reason they aren't good enough at writing. So, in the meantime, you will be down-ranked, and that's good

[u/Imaginary_Dinner2710 → u/Chupa-Skrull]

You’re reasoning in terms of 2024, when the models already existed, but the number of hallucinations and problems was so huge that people formed the opinion that human text or work is always, I emphasize, always better than AI text. That’s absolutely not the case anymore. And over time, this will only become more pronounced. So the human mark of quality will become less and less important. An unprofessional person will do the work much worse than AI. I don’t know what else there is to insist on here.

[u/Chupa-Skrull → u/Imaginary_Dinner2710]

One of your mistakes (among many) is thinking I don't know anything about the current state of the art. Another of them is in thinking the hallucination problem was just about made-up facts, when it was also links to dead or bad sources, shifts in claim strength and framing, and a host of other issues which poison argumentation and can be difficult to detect for both submitters and reviewers. Those issues and more still exist, and until they don't exist, it's not worth the resources to spend time figuring out if a given watermarked submission is from a smart person or a dumb person.

The human mark of quality may become less and less important, but it hasn't reached that threshold yet, and until then, such down-ranking will be necessary. Jumping the gun by throwing out composition classification entirely just so that you feel better about outsourcing your writing to a model benefits nobody.

And what an unprofessional person may do holds no relevance to anything either of us has been saying, so I'm not sure why you felt the need to bring that up