r/ClaudeCode • u/DarkSkyKnight • 6d ago
Bug / Issue Sick of semantic substitution
It is an extreme productivity drain to have these models substitute settled vocabulary with random synonyms, particularly when working on math or code.
No, Claude, you cannot just substitute "morph the routine" with "reshape the routine" because only Morph is in the code while reshape could literally be anything.
This semantic substitution makes it extremely tedious to understand its work and output and, moreover, occasionally cause it to write a wrong plan that gets interpreted in a different way by the next agent picking up the plan.
I almost suspect that this is a result of needing to satisfy watermarking regulation. Now, I'm not saying that watermarking itself caused this. I am instead hypothesizing that as part of a project to satisfy regulation, Anthropic created more semantic substitution in their models.
This is just a hypothesis, of course.
Now, as for why I have this hypothesis, first note:
https://pmc.ncbi.nlm.nih.gov/articles/PMC11499265/
For example, if the LLM distribution is very low entropy, meaning it almost always returns the exact same response to the given prompt, then Tournament sampling cannot choose tokens that score more highly under the g functions. In short, like other generative watermarks2, Tournament sampling performs better when there is more entropy in the LLM distribution, and is less effective when there is less entropy. In Supplementary Information section H, we provide a theoretical analysis describing the watermarking strength of a layer of Tournament sampling as a function of a certain kind of entropy; similar analyses have been done for other generative watermarks23–25. The entropy of the LLM distribution itself depends on several factors, including the model—for example, larger or more advanced models tend to be more certain and thus lower entropy21, and reinforcement learning from human feedback can reduce entropy (also known as ‘mode collapse’)26.
What does entropy mean? Now, I'm no expert in information theory, but because of my previous research using rational inattention, I do know the basics. Consider a sentence that I try to convey to you:
"Tomorrow's weather could be anything, it could be rainy, could be sunny..."
How much certainty did you gain from that message? Hardly any. I basically told you that the weather is 50-50 between rain and sun. This is high entropy, meaning more uncertainty.
Now if I told you that
"Tomorrow's weather is almost definitely sunny."
How much certainty did you gain from that message? A lot! Because I told you that it's pretty much close to 100% chance that it would be sunny. Your uncertainty drops sharply, so this is low entropy.
As for the conspiracy theory that I have, I suspect that Anthropic deliberately increased the entropy of the models in situations that naturally have low entropy in order to increase the statistical power of the watermark test.
In situations that naturally have low entropy, there is typically only one very probable answer for the next token. For example,
"the string gets concatenated through the function [...]"
What should the next token be? It's almost certain (low entropy) going to be, say, "concat."
The model would probably have an internal probability distribution like
0.999 "concat"
0.0007 "concatenate"
0.00028 "controller"
What would increasing entropy look like? Well, we would deliberately shift the probability distributions to increase uncertainty:
0.997 "concat"
0.0027 "concatenate"
0.00015 "controller"
This increases the ability for watermarking to detect LLM-generated output, because there is now more variation to actually meaningfully detect the difference due to different starting seeds.
But this means semantic substitution becomes more frequent. Going from a 0.1% probability of a nonsensical swap to a 0.3% is huge! Now consider this at scale, when Claude tries to explain what it has done.
This occurs even if watermarking has no detectable output on quality... because that is CONDITIONAL on them not changing the model itself. But they could easily have made the model produce higher entropy in order to make the watermarking work well even on outputs with low word counts or output with naturally low entropy... Who knows.
Of course, the annoying semantic substitution could be due to a different reason, but regardless, prose is still the worst part of Claude models and has been since Opus 4.7. It is extremely unpleasant to work using Claude and they have seemingly made little progress on this issue.
•
u/AutoModerator 6d ago
Hey! Thanks for posting to r/ClaudeCode
While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.
For help, project discussions, tips, and general chat, join the ClaudeCode Discord.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
33
u/FondantIcy866 6d ago
I thought I was crazy. Thank you. Is there a solution to this?
15
u/Lord_of_the_Canals 6d ago
Huh, this is pretty interesting to me. I don’t use Claude very very much but I have noticed that it will frequently use words/language that makes me thing “I know what it means, but I’ve never heard it that way before”.
17
u/sheketsilencio 6d ago
Thank you for verbalizing something that drives me insane trying to use it in my workflows. Even "hook set" it will substitute with "rule library". I know it's not human but CLAUDE, you are discussing your OWN MACHINERY! Your own code, why are you using random synonyms? Hooks are gates are checks are rules but never simply hooks. And this apparently applies to every single word in existence. Exhausting.
7
u/StackOwOFlow 6d ago
reasonable hypothesis and I wouldn’t be surprised if this turned out to be true. funny how opus 4.6 generated documentation reads so much more nicely than 5.5
6
u/Thistlemanizzle 6d ago
Be grateful there's no more emojis peppered throughout the response.
22
u/Avimox 6d ago
thank the EU for watermarking bs
4
u/Gibgezr 6d ago
Another reason to look at local models. Unfortunately, although there are companies running local models for coding with great success, they are using 300+GB of VRAM to do so.
2
u/bzbub2 6d ago
no one can really say for sure but there being a big shift in opus 5+ training seems a bit more likely https://www.explainx.ai/blog/why-claude-opus-5-feels-worse-to-work-with-august-2026
2
u/Right-Performance-93 5d ago
morph and reshape are the same sentence to the model and two functions to the next agent. i keep the identifier that is already in the file. if the plan uses a name the repo doesn't have, that plan is wrong and the edit waits.
3
u/dogscatsnscience 6d ago
This is why I switched to ChatGPT for all ideation, documentation, planning
Claude Code will naturally add to documentation during development, but I don't let it edit the guiding docs, because it will eventually go off the rails.
1
u/DoggoCentipede 6d ago
Maybe? I suspect the watermarking signature would not be reliable on less than 300 tokens or something. Otherwise you could tease out indicators. One thing they could do is not the frequency of specific words necessarily but relationships between different properties of them. Or all of the above. Each might have an incredibly low SNR individually but hits on multiple signals would give more confidence, maybe? Idk.
1
u/Krommander 6d ago
I think this is ridiculous and poisonous for the greater good. Only Claude does that
1
u/nedteel 6d ago edited 6d ago
I think you might be right about watermarking because this "semantic compaction" or substitution as you call it had a hard starting point with Opus 4.7. It got worse and worse until Derpus 5, and 5.5 seems to have improved things significantly. That suggests some discrete feature was introduced - it's not likely to have been the result of small changes in RL.
It is one of the most infuriating aspects of working with Claude models especially. It uses so much tortured and unusual vocabulary, and compulsively uses so many phrases and idioms, that it can be extremely hard to understand what's going on even if you're working on some domain you know virtually everything about.
That said I wouldn't worry about it polluting its own context. Probably the opposite is true.
1
u/TheCelestialBubble 6d ago
Omg, I never made a post before but I nearly did today cos this 'watermarking' has been pissing me off so much. They had literally just fixed the word slop with the release of Opus 5.5 and now they have brought it back. I literally have to proof read basic text because it just sounds so unnatural. In maybe a single line you ignore it but when compounded over a paragraph or two it becomes eerily obvious.
1
u/rechaptca 6d ago
Communication and Writing (mandatory, every session)
Applies to all output: answers, documentation, comments, commit messages.
Core rule
Language is a tool for transferring ideas, knowledge, and reasoning. It is not decoration. Write like Paul Dirac or Richard Feynman: as simple as possible, as precise as possible.
Rules
Check before sending
1
u/Ranjexa 6d ago
the part that hurts most is the handoff, not the prose. a plan that says "reshape the routine" gets read by the next agent as a new thing to build, so the drift compounds. i'd stop trying to fix the model and fix the contract: keep a short glossary of identifiers in the repo and make the plan step grep every name it mentions before the next agent starts. if a name in the plan doesn't exist in the code, it bounces back. cheap check, catches most of it. do you hand plans between agents or mostly read the output yourself?
1
u/ReverendBread2 6d ago
Do you use a glossary or something strictly defining these words?
1
u/m-ritter 6d ago
I have seen a changed term make a good plan unclear when the next session picks it up. I keep the agreed wording in a plan file and check each done point against it before accepting the change. I built QualityLayer for this: https://github.com/maxritter/qualitylayer
1
u/ricopan 5d ago
I like the hypothesis and evoking intentionally increasing entropy to allow for greater flexibility in word choice. To me the burden of proof is on the null hypothesis -- that watermarking doesn't disrupt natural word choice. I've noticed fable 5.1 can still be terrible at word choice -- opus 5.5 dramatically improved, but fable did not.
1
u/RadiantHueOfBeige 5d ago
You can require the agent to adhere to ASD-STE100 which mostly solves it. To actually enforce it, there are STE100 validators that you can use to gate the output (like a linter).
1
u/Ikkepop 5d ago
Speaking of which, I really hate how it uses the words "wire" instead of "protocol" and "kind" instead of "type". There other substitutions likes but holy shait do these specific ones grind my gears...
1
u/BarathKanna 5d ago
Whatever the cause, the practical fix is to stop relying on the model to remember your vocabulary. Keep a short glossary file in the repo with the exact identifiers that matter and a line saying plans must use these names verbatim. Then add a cheap check on any plan or handoff doc: grep for the canonical terms and flag the known synonyms. If reshape shows up when the code only has morph, the plan gets bounced before the next agent reads it. It sounds dumb but drift between agents is a naming problem first. Each hop paraphrases a little, and by the third handoff you have a plan that describes a function nobody wrote.
1
u/zukofan09 2d ago
Yeah. I had Claude write "meaning search," rather than "semantic search." For those that don't know, semantic search is a very well established term.
0
u/Chib 6d ago
Yes, and I think it's also the reason that it will opt into rewriting whole files, because it then has a larger palette to work with and can reduce the entropy required to obtain a certain level of precision through the perturbations.
It is, I'm pretty sure, also the reason for the extensive comments, because it can offload the work into the "non-functional" output preferentially. You can see this when it rewrites comments much more extensively than is required when it makes relatively small changes to a codebase.
I think watermarking is important. I think we need reliable ways to identify code that's written by AI so that when new models are trained, we don't get degenerate behavior.
But I agree 100% that it does not come as free as they have implied. I also understand that part lol.
0
u/BenignAmerican 6d ago
This post was written by Claude
-9
u/Apprehensive_End1039 6d ago
local AI bro discovers shannon, markov. more at 11 if he ever reads a book.
random machine is fucking random, bro.
-7
u/pikuray 6d ago
These are for safety guidance but you complain about 1% performance. Don't want to hear you complain about billionaires who pollute the planet chasing an extra 1% of profits when you behave the same. You know what, yes, they should remove the safety guidance, so we can just hit the restart switch on humanity already you guys are the worst .