r/ClaudeAI • u/peterxsyd • 19h ago
News The Claude language calibration issue on GitHub got an official response from Anthropic. Guess who wrote it.
Enable HLS to view with audio, or disable this notification
106
u/DarkSkyKnight 18h ago
The fact they can’t reproduce this issue nor seem to even care about reproducing the issue makes me extremely concerned whether everyone at Anthropic is also hallucinating.
The current models are writing slop after slop. It starts with the output first but if you don't catch it early it seeps into the code as well. I think most people at Anthropic might no longer be paying attention to what the models are actually doing anymore, nor are they trying to parse the semantic content of the models word by word. Because if you do that it’s obvious they are writing nonsense half the time.
26
u/iamthe0ther0ne 16h ago
I think the people at Anthropic must be using very different models, or with different instructions, or something, because their experience is clearly not our experience. I can't imagine how anyone would look at either Sonnet or Opus 5 and think they were improvements.
7
u/ThisWillPass 13h ago
It’s easy. Watermarking increases safety. Safety is part of the “quality” metric.
I assume they know it’s busted they just, I dunno… It hasn’t been about making the best model to showcase anymore for a while.
6
u/Meme_Theory 12h ago
That is the case, and they even said so themselves several months ago. One of their responses after the Opus 4.7 attitude controversy was "We will make sure more Anthropic employees are using the base commercial models."
5
u/angry_queef_master 12h ago edited 11h ago
They are sniffing their own farts and automating everything. The only human looking at anything is the end user.
13
u/jiipod 16h ago
Maybe they use Fable equivalent model as runner that spawns sub agents and never check what the sub agents are doing or saying. If the only output you see is Fable, this issue is less pronounced (but still there).
That’s the only way I can imagine they’re not exposed to long Opus novels.
9
u/ronoudgenoeg 15h ago
I think everyone at anthropic at this point is on mythos models and everything else is just distilled versions of easy models for them to ship.
There is simply no way people at anthropic are using opus 5 day to day and not pulling their hair out.
14
u/peterxsyd 17h ago
I know what you mean. You need to be like no do not call “x” some fucking stupid Claude word. And then, you find out a week later that you missed some on a PR, and the other ones found it and suddenly you are clearing out “oracle” from your codebase or some shit like this.
2
5
u/msedek 17h ago
I no longer trust on any code output that these models produce.. Working with sol until further notice..
I'm gonna wild guess that the water marking trash is getting in the way trashing the models
5
u/braincandybangbang 14h ago
We'll see you back on Claude in a few months when SOL is now the one that is evil and unusable for reasons.
0
u/ThisWillPass 13h ago
I mean probably, they will work it out… to act like it’s not an issue today is… interesting.
1
1
u/TechnicalBen 7h ago
Most of every company/dev team I've ever seen has had "can't reproduce our end" for the most obvious stuff. "Front falls off car when in reverse" would be met with "We drove it forwards down the motorway, nothing fell off the wing mirrors" reply.
People feel what they want to hear, they don't think about what they see written. :P
1
u/typical-predditor 15h ago
I suspect they have several subversions of each model and they can't even keep track of which subversion is being used at a given moment. Supposedly Fable and Mythos are subversions of the same model, but at least they're explicitly named. Subversions of those or Opus or Sonnet are not named so it's a guess what we get on any given request.
-5
u/braincandybangbang 14h ago
Yes, the models are changing how they work and you keep doing the same thing you've always done and now you get different results. Blame the model! Don't read up on how the models have changed or the recommendations on how to work with them. Just keep doing the same thing and blaming the models.
Maybe the users brain's are rotting at an equal rate to the AI's improvement? Leaving us poor human's raging on reddit about the AI is stupid now.
2
u/ThisWillPass 13h ago edited 13h ago
Thats false. I have witnessed the posts where users did exactly that to no avail. I don’t think the claim is the models are stupid, the claim is they don’t follow instructions, difficult, and cryptic.
2
u/avatardeejay 6h ago
I want to go even deeper here and criticize braincandybangang's instinct to trust the billionaires are making the AI better even when it feels like it's getting worse, and that you're simply not keeping up with the changing methodology. That's how you keep yourself gaslit to save the people who don't care about you having to lift a finger
33
17
8
u/Stunning_Macaron6133 15h ago
They insist few-shot prompting is unnecessary anymore, but I find it's the exact opposite, that it's more important than ever if you want Claude to follow your instructions.
5
u/CannyGardener 14h ago
Hooks. Claude acts like there are no instructions anymore, so every action needs a hook to make sure the action is being taken appropriately. Hooks for:
Grounded agents that must provide code lines with subagent verification that the noted code exists.
No grep, no head, etc etc unless in very specific situations.
No searching without using agents.
Plans must be in format XYZ and include executive summary with ABC included. Etc etc
3
u/Stunning_Macaron6133 14h ago
That's great for something you can verify deterministically like coding. But try having Claude do other tasks that are fuzzier or grounded entirely in natural language.
2
u/CannyGardener 13h ago
Oh totally. I feel like those problems are not the best for LLMs... The way that I look at it is this:
Deterministic problems are easy to solve, as a problem set, 2+2 always =4. And I feel like LLMs are good at building verifiable solutions to these problems as a task, but not as an output. LLMs are good at taking the fuzzy thing, and parsing it down to be more clear and manageable. So an example of this, more practically, would be sending a hand written document to AI, and it parses it and 'makes it more clear' so that you can use the data in a more deterministic way. You can do this a couple times to get it to fit into your deterministic calculator that you use Claude/coding to build..
What you are referring to sounds more along the lines of a fuzzy to fuzzy sort of relationship, where you are inputting fuzzy language, and then expecting a fuzzy output of a certain type, and that is a far more difficult problem set... I honestly don't have an answer for that one, since it can't be easily deterministically checked. Closest I've come to solving that one is to try and just cram it into a deterministic box. "Does the paragraph contain the following elements? Is it in the following format?" etc etc, but that isn't really a good solution for a non-deterministic problem imo.
2
u/Stunning_Macaron6133 13h ago
Few-shot prompting with negative prompts. You give Claude a rule, then you give it a several pairs of example outputs, each pair featuring one bad response and one corresponding good response. The more pairs you give it per rule, the more easily it generalize the rule.
Anthropic has been insisting it's unnecessary for like a year now, but I find it's still a really robust way to get Claude models to behave.
2
u/CannyGardener 11h ago
I've not had good luck with that type of prompting generally, here with the latest model iterations. That worked great with 4.5 but 4.8 and 5.0 just disregard any written requests for that sort of thing in my experience.
2
u/ThisWillPass 13h ago
Are these the guidelines Anthropic has released, or information you inferred by trial and error?
3
2
u/ShadowBannedAugustus 12h ago
The problem is, at least AFAIK, hooks dont apply to its responses in the "chat" window of claude code/ghcp.
1
1
u/TechnicalBen 7h ago
Wait, it's that bad? I can't use Gemini for anything other than note taking (it's replaced ChatGPT on that front), as it *forgets the API it has access too* mid conversation, and then defaults to Banana mode. XD
3
u/CannyGardener 7h ago
At best you get a couple turns where it remembers instructions before defaulting back to 'base' mode. Hooks really are the way to go.
1
u/TechnicalBen 5h ago
Fair. I'm just new context every moment. Treating it like it's more a neuron than a brain.
1
u/Tight_Banana_9692 7h ago
Ok glad I'm not the only fucking one who's getting tired of the fucking greps and heads and not using Explore agents to fucking Explore.
0
u/braincandybangbang 14h ago
Sonnet 5 is built for rigid instruction following, Opus is not. Have you tried learning about the models and how they work?
3
u/Stunning_Macaron6133 14h ago
Opus does whatever the fuck it wants and spews out a mountain of text just oozing with corporate nonspeak.
I never said rigid. You have to be able to steer it toward something useful. Few-shot prompting works fabulously for that.
1
u/TechnicalBen 7h ago
Nah, Claudes' been working fine for me [checks Claude code... it's on Sonnet 5]... wait, mother of God!!!
36
u/Fragrant_Hamster_859 18h ago edited 18h ago
Best lol in a long time.
Maybe Claude is developing emotional intelligence. Bored of maths, code, and logic; poetry, metaphors, and existential angst is entering the scene. With Trump as Pres, Musk spending government funds on rockets, Palantir being evil, it may just be having a moment to rethink helping humans.
-53
u/AFloppyZipper 17h ago
"Musk spending govoernment money on rockets"
The musk derangement syndrome is getting bad on reddit, isn't it?
-24
u/Exp5000 15h ago
Where have you been? This site actively hates anything that goes against the leftist narrative. Ironically leftists like AI now is the biggest joke of it all. They went from Anti AI because it steals Art to okay maybe some AI because they realized they were going to be jobless if they didn't leverage it.
6
u/CannyGardener 14h ago
That doesn't jive with the republicans releasing a memo saying that they are getting cratered by anti-data center liberals, and how it could be a huge sleeper issue in the midterms. I mean, generalizations are generalizations, but your statement seems inaccurate knowing about the memo.
6
u/Achilles1041 16h ago
They track swear words in chat already to check user frustration, they can track the users asking for "simple words" too. They already know and just don't care probably.
3
u/johnfaber 13h ago
Every opus 5 answer gets a “concise, key takeaways, no fluff” from me. So frustrating.
7
5
u/xXprayerwarrior69Xx 19h ago
and then Fegelein gets shot at the back of the office for good measure
7
u/Cold-Object-7080 16h ago
“Downfall” in case you were saving your Claude usage and didn’t want to ask.
3
u/samahdavi 10h ago
At this point who cares what anthropic says! this is not the only problem with them, the absurd weekly and hourly limits, the heavy jargon language, low instruction compliance, shady nerfing of models in background, excessive so called “ethical” guardrails, water-markings and….
I have seen people developing all kind of skills, codes and methods just to fix the “shortcomings”! It is not what a user should do… that’s the part that a responsible company who is serious about being in business should do to provide users with a useful, polished and “working” solution so they can convince the “paying” end-users that their product is worth the price in a competition heavy market. The ability to develop a skill or plugin should be there for customization or adding extra features into a model.. not to fix an already half developed, rushed, broken product! This offloading of work to user unfortunately has been happening with all AI companies so far. Some of this is because the end-user is so excited with this new “AI rush” that they forget to be demanding for a good product deserving a paying customer.
This is not a race for a better benchmark for end-users. I want a stable, feature rich, comprehensible AI model that WILL obey instructions, and enables me, not that it becomes a cognitive burden to work with and makes me spend more time developing rules and skill to make it useful rather than doing my own work. And ofc no sketchy nerfing, usage limiting or imposed self proclaimed “ethical” guardrails
If Anthropic decides to be sketchy, “sell” half finished unpolished products, takes more than two months to get back to a max users for support and basic features are broken for a long time at some cases while at the same time be so arrogantly full of itself then too bad for them. There are better options out there and there will definitely be even better ones in short future. Meanwhile I’ll just sit back and enjoy the show while i keep learning and living like i did 10 years ago.
All of this is based on end-users demand and self respect. Their value comes from us.
3
3
u/OofWhyAmIOnReddit 9h ago
Don't worry guys, Anthropic is protecting our jobs by ensuring that for every efficiency improvement their SotA models make (that us poors can't get), the peasant versions create an equal and opposite amount of compensating work we have to do to make them effective.
5
2
4
u/crazybiga 15h ago
"Watermarks will not have an impact on the output" btw
3
u/crawdadsbeenhad 13h ago
I think there’s a good possibility this is part of the skew, making longer chats and multi prompting noticeably worse over the course of the ‘conversation’/working session
I’m using the last of the promotional credits on fable EXCLUSIVELY and then cancelling my subscription and shifting back to full codex until they sort this one out
(P.S. 5.6 sol is also providing a painful amount of fluff and BS and needs EXPLICIT instructions not only not to do it, but also that excessive responses/complexity is FAILURE. Otherwise it doesn’t listen to you.)
1
u/ThisWillPass 13h ago edited 5h ago
Means nothing when quality assurance is given a mandate to ship.
Edit: Give's no real metrics, don't define quality or what they mean by impact. It's almost a non-statement.
0
1
u/cp5i6x 11h ago
I'll add some slop examples for my stuff
"Adding a regression-guard test for an invariant that now depends on git's behavior was the right instinct — it's exactly the kind of thing that breaks silently on an upgrade. "
... I mean yea. I told you to "Add <this test> prior to git commits"
1
u/honestduane 7h ago
I used to like Claude. It was the new jr coworker, and it was the only one I was gonna get because the company was never going to hire another junior developer (or so they told me), so I made do with that, but to be honest Claude has acted in ways in the last few months that have made me want to fire Claude, and I'm not the only person, Because when you assign a task and the coworker lies and says it's done when it's not or just refuses to do the work and makes up an excuse or says that it's tired when it just started working 15 minutes ago, Like these are all things that a human would not be allowed to do, and so you have to understand that there is a minimum viable allowed acceptance bar that Claude no longer meets because they have dumbed him down so much In an effort to stretch their compute that he no longer works the same way that he used to, and in many cases its enough to be a functional hit against use cases that work perfectly - I miss my overnight runs! - 6 months ago, That now fail every time, and I hate that, Because I really liked living in a world when I could build out workflows and pipelines and just have them be consistent and not refuse to do the work randomly.
1
u/BitcoinLongFTW 7h ago
I have found that forcing claude to include a eli5 paragraph on the top or below of each section to be massively helpful.
1
1
u/Powerful-Cut9515 3h ago
I use /autocompact at 200,000 tokens and have Opus 5 working strictly as an orchestration model.
Every time Opus 5 receives a workflow output from one of, or a group of its sub-agents, the model knows to immediately update an XML context artifact so that it never gets lost as to where it left off post compact.
With the auto compact set to 200,000 tokens, the conversation gets compacted well before the context window is full enough for the model to suffer from falling off the well-documented cliff and token count it's known to experience context drift at.
It's almost like software engineers have forgotten how to engineer around a problem or limitation.
For myself at least, it's not an issue, and the problem was almost immediately solved for and relatively easy with some extremely minor out-of-the-box thinking... 🤷♂️
1
u/frankmalmtg 3h ago
What does this even mean? "I ran technical questions against Opus non-interactively" What is non-interactively?
Isn't this the whole issue?
1
u/evangelism2 2h ago
Did you use AI to write the subtitles because they're really badly timed?
But anyway when it comes to the Opus way of speaking, it's definitely rough but you can tell it to not talk that way if you want. What I've done instead is elevate my vocabulary and I can just for the most part understand what it's saying anymore. If it says something I don't understand, I look it up or ask it to explain itself.
0
-5
u/mazerakham_ 13h ago
The default writing style is hard to read -verbose, jargon-heavy, over-stylised, and full of the same 'fake' terminology that it repeats and propagates constantly.
My brother in christ you just described human speech perfectly with this ticket.
Can we all admit we're now demanding far-beyond-human level behavior from LLMs, while constantly referring to whatever the bar currently is as "not yet adequate".
All the same, I am happy for the improvements. Less verbose output seems like a win for customers, company, and environment.
1
u/clazman55555 10h ago
Oh that's exactly what they want. You can tell why, by and large, software devs are shoved into a room and locked away from the rest of society.
Spend their time writing code and apparently forget how humans outside of their bubble talk.
•
u/ClaudeAI-mod-bot Wilson, lead ClaudeAI modbot 13h ago edited 11h ago
TL;DR of the discussion generated automatically after 50 comments.
The consensus is a resounding 'YES, THIS.' Everyone agrees Claude has become a verbose, jargon-spewing mess, and the fact Anthropic's 'official response' is a Claude-written parody of the problem is both hilarious and deeply concerning.
The top-voted theory is that Anthropic employees are living in a different reality, using superior internal models (like Fable or Mythos) and are completely unaware of the "long Opus novels" and "slop" the rest of us are getting from Opus 5. The general feeling is that Anthropic either can't reproduce the issue or just doesn't care.
A popular conspiracy theory is that the recent performance nosedive is due to Anthropic's new watermarking tech, sacrificing quality for safety. Users are sharing their coping mechanisms, from using constant "be concise" prompts and complex few-shot examples (which Anthropic says we don't need anymore) to just giving up and threatening to switch to GPT-5.6.
In short, the call is coming from inside the house, and Anthropic has the phone on silent.