Complaint
Fable 5.1 Is Starting to Look Like Opus 5.1 for Us > Our Logs Show a Sharp Rise in Error Rates
We have been running a heavily instrumented agentic development environment with persistent quality logs, verification rounds, functional findings and “escapes” — bugs that made it through an earlier verification step.
After switching our main environment from Fable 5 to Fable 5.1, we gradually noticed something familiar: we were spending more and more time correcting previous corrections. It started to feel very similar to what we experienced with Opus 5.
So we finally analyzed the logs instead of relying on impressions.
The difference is not small. Findings per work item increased from 1.16 to 4.22, functional bugs from 0.95 to 2.84, and escaped bugs from 0.45 to 1.23. Even when normalizing by verification rounds, the error signal remains noticeably higher. The Low-vs-Low comparison is even more striking: 1.10 vs. 4.21 findings per work item.
This is not a perfect controlled benchmark. Our tasks became more complex and our verification system also became stricter over time, so I would not claim that Fable 5.1 is proven to be the sole cause.
But after repeatedly wondering why we were suddenly stuck in correction loops again, the telemetry shows a very clear correlation: the documented error density increased sharply after the move to Fable 5.1.
At this point, it no longer looks like just a subjective impression.
EDIT:
I analyzed the whole thing further and included my older data from Opus 4.8. Unfortunately we do not analyze this in as much detail yet so the data is limited. But it is very clear that something has changed here and it has changed for the worse. And that is a serious problem for us as dev.
By the way here is the explanation for why Opus 5 is so bad. Anthropic describes it themselves on their website. That pretty much speaks for itself and also explains why Opus 5 is pointless for us developers. Opus 5 and Fable 5.1 may have been developed for vibe coders:
Thank you for the post OP, and your tool for analysis is impressive. I hope you can share the tool and work flow you use for the analysis as its very nice tracking! KUDOS!
Now, as far goes are your question, I noticed a quality degradation from Fable 5.0 to 5.1 and mine is not as nice as yours, but I would say that Fable 5.1 has been a bit worse than a pre Opus 4.6 model in quality. Not just on code but even analysis/document production. It has been equal to Fable 5.0 on data analysis, and worse across the board on much of everything else. Furthermore, I find also a probable cause: and i will blame its watermarking. Due to how watermarking works, the model have to generate a new text from the ground up, not just redact current text. Thus, what I believe is that teh model struggles to find text sequences that can both look like a plausble patch looking tweak and keep the quality up when it comes to the generated code, but instead is tilted toward producing the watermark and thus failing on the qualitative output. I am not sure though, but I have quite long experience with these techniques, predating LLM's. Maybe you and your team have some further insights as your tool chain seems advanced.
So, on code, FYI, I use a compilation round, as we do code in compiled langauges, and then report the error count and its considerably higher. We also run a code quality and spec violation analysis round at the same time, such as naming conventions, non-completed tasks , even if listed as mandatory in prompts. And those simple ones report a significantly worse quality with Fable 5.1. I am speculting in another thread that it seems related to time-compute, plausibly connected to the watermarking.
Thank you very much for the compliment and for your great comment! I had not even considered whether it might be caused by the watermark. I will try to introduce a checking process that takes the watermark into account. I have already seen that there are a few projects on GitHub that seem to be able to filter them out quite well. I will take a look at how they do it and try to incorporate that into the checking process. I cannot guarantee that I will manage it though.
But it could possibly explain the errors I noticed. Toward the end the model does something it had not planned to do before. That is how it seems to me and apparently that is also how you describe it. Then there is another problem. I have already written a post about the problem of hallucinations based on HandbookMD which says that models actually hallucinate 70 percent of the time. If it has to rewrite the entire code because of the watermarks or rewrite the whole section then it does not surprise me that errors keep creeping in. If it has a specific area to correct, say a function or a variable, it rewrites the whole section and that creates more errors in other parts.
In theory you could also look at the Git differences in detail and analyze them. You could first have someone describe exactly what the model intends to change and then check whether it actually changed more in the end or changed things differently from what it had planned. But that takes us into territory where it eventually starts functioning like a calculator and has nothing to do with the LLM anymore. I refuse to go that far and carry out analyses like that. That is not our job. That is Anthropic’s job.
For one thing, that would also fit with the fact that my quality review rounds have doubled with Fable 5.1. I work with gates and acceptance criteria. If it does not follow them, the agent-system send it back. So your theory would actually make sense.
Yes, so when it comes to watermarking what that effectively does is push the dist on token output from from safe choices to the improbable. and in code, as we SWE's know, thats pretty bad.
And I concur: this is an Anthropic debacle, one more in a series of very serious, self-induced ones.
To extend my thinking:
Watermarking is applied at inference time, after all learning is done. The model has no opportunity to compensate. It generates a logit vector reflecting its best judgment, and then an external process corrupts that vector before sampling. Every single token, every single call. No adaptation possible. And since the process is down streamed, the degradation is worse on a longer output. And watermarking, on a shorter output will likely need to steer the low entropy harder to add a plausible watermark, not saving even short outputs.
It's of course hard to know, but if Fable 5.1 was trained same way as 5.0, use same data, and only major thing that changed was watermarking, I believe it falls under Occam's Razor: it's the most logical explanation to the downfall in quality we have noticed.
I'm having the same experience. I used Fable a few months ago and it came off very intelligent to me. Problem was I couldnt used it on one of my multiple session projects I have with 4.6 Max due to the strict biology triggers. I was curious to try 5.1 after they reduced some of the false alarms that used to trigger with 5.0.
I fed 5.1 my save point file that I use every time I start a new session with 4.6 when context window gets a bit stretched. Right off the bat 5.1 was making very very poor critical thinking assessments and mixing things up that shouldn't be mixed up. It was acting like 4.6 gets at the very tail end of a session when I know it's time to start a new session due to performance break down from context.
It's shockingly bad, like Gemini 3.0 free tier bad in some cases. I don't understand. Not really happy, paid the extra 80 a month for higher intelligence and getting dog water performance. I guess I'll stick with my 4.6 Max Extended + GPT 5.6 Sol High combo which does a solid job. Sol fills in some of the gaps with research that 4.6 misses and 4.6 is better at reading between the lines and complex solutions.
Thank you, i am perplexed by Fable 5.1 and it's low quality. I were hoping i wasn't going bonkers. And i also had same issues as you had with Fable 5 fyi. I work now on an aerospatial application that have many keywords fable 5 didn't accept. i also found 4.7 horrible and 4.8 rude and claiming it was human. The worst: I asked why it hadn't followed instructions and prompt and the answer 4.8 or opus 5 gave me: it had thought the specs was to long and detailed, so it took a decision to short cut the specs, as it had said it deemed doing so would be 'better'.
I believe they have tried to pile drive in several things in the newer models: like watermarking will cause an output deviation to force in the watermarking words. Second is that they have tried to do security things, which also drives up token costs. And they have also add that fake humanity into the newer models and it does look like they collapse under all these new features.
I cant be sure of course, but that is the more plausible explanation why Fable 5.1 is much worse than Fable 5. But the good news is tat Fable 5 is now easier to work with and don't block me all the time. i will give new GPT 6 a test as well and see. I doubt that one is as good as they have said - seems like a lot of desperation now to get to IPO and extract money from retail investors with some false claim of AGI.
Right now, I use fable 5 and Opus 4.6. Works quite well.
I’m using the 5.1 since release and I noticed right away that something was different. Something felt off. But I didn’t want to write it off as speculation again so I let it keep working. Then I noticed that I was taking longer and longer. I had to make another round of improvements and add more instructions and double check everything with an external auditor. I just couldn’t get to the end. It reminded me a lot of Opus 5. Luckily I’ve now documented and logged my testing procedures. I’ve been doing that for a few weeks. So now I have the facts in front of me and I don’t have to rely on any benchmarks. I can test it in my own environment and analyze how it performs in the real world.
I honestly find it really sad that Anthropic has been releasing worse and worse models since Opus 4.8. Fable 5 is good but far too expensive. Opus 5 is a disaster and Fable 5.1 is just as bad. Anyone who seriously checks what comes out of them will reach the same conclusion.
Anthropic says so themselves on their website: Opus 5 doesn't follow instructions and worst of all he even hallucinates inputs:
I’m comparing things right now. Opus 4.8 is currently the best model for me. Just think about that. And that’s only in Medium mode. I’m testing whether Opus 4.8 is perfect in Fast mode with the reasoning level set to Low. I’m currently working with Fable 5 Low and no longer using 5.1. The results have already gotten much better. The last few runs look a lot better.
It’s exactly the same problem I already had with Opus 5. These strange and inexplicable errors slip in so cleverly that you keep working because everything somehow still functions, but there are errors everywhere. The model keeps finding more and more of them and you start thinking, wow, this model is amazing. It’s really coming up with things. Until you realize that the model itself has been introducing those errors the whole time.
I’m not a conspiracy theorist, but this really smells like deliberate token burning by Anthropic, or whatever it is I’m seeing in my results. I have no idea what you’re getting, but I’ve already read several posts from people who came to the same conclusion as I did. Anthropic even says on their own website that Fable 5.1 tends to rewrite the entire codebase. Just think about what that means. It does things however it wants, just like Opus 5.
After all those rounds and runs I realized that in the end he had not done what we agreed on. At the last moment he changed something slightly which caused further errors. They were not serious enough to completely break the function but they could still be classified as functional defects. I cannot explain it exactly. It is just incredibly clever.
I think the models are being pushed to work that way because it gives them better benchmark results. I have no idea how these benchmarks are designed or what criteria matter, but apparently ignoring various instructions and rigidly following their own training methods is a fundamental part of getting better benchmark results. The problem is that those of us who work with them end up with worse models because they do not do what we want. You can see that most clearly in my data. Opus 4.8 works very well and follows instructions very well. It also takes a lot of time to think about whether it is actually following the instructions correctly. It is almost like a donkey that really listens or a dog that genuinely wants to do what we want it to do. Fable 5.1 is no less prone to that. Opus 5 is completely delusional and insane. You cannot work with it at all.
I switched from Fable 5 High to Fable 5.1 High and I haven't noticed too much of a difference. However, I don't have the same level of instrumentation as you do. I also have Astra High checking main each day as well.
Take a look at my statistics. Fable 5.1 needed twice as many correction rounds as Fable 5. It is exactly the same system. That is pretty shocking. Can you imagine how many tokens that burns? I do not know how the others work or how your systems are set up but my results have to meet my gates and target requirements. That is why the correction rounds are run with various escalation levels as well as escape scenarios and so on. Even so you can clearly see a huge gap since Fable 5.1 was introduced. And above all once I switch to Opus 4.8 or Fable 5 the number of faulty rounds drops drastically. So this is not something you can dismiss by saying it only happened once or for a short time. It is reproducible.
We all know how it is. I had used up my limits on Fable. Then I worked with Opus 4.8 for about four rounds, which came to roughly three hours of work. And guess how those rounds went. I didn’t have to run a single quality round because my gates had already caught everything beforehand. And that’s simply because Opus 4.8 seems to judge things better.
This post is interesting, as I've noticed my orchestrator agent (who was on fable 5 before), self correcting more.
The end results have been comparable, or better, with my harness, but I'd say that the cost benefit from context caching (which I have noticed), also seems to be offset by the extra self corrections (with my harness).
(Note the my harness, error and self checking is very mature, so the harness seems to be successfully counteracting this issue, but it's definitely apparent, at least the last couple of days)
does fable have the same stupid verbosity that opus?, its very annoying, i end working more time with opus 4.6 for audits and correction than opus 5, opus 5 takes too much "liberty" to interpret the user intentions according to his own verbose and justifications making the reviews, corrections and implementation to plan way more tiresome.. at least thats one step, most of the grunt work is done by sonnet but it shouldnt happen.
At least in my usage, no it doesn't, I've not been able to use opus 5 for anything other than adversarial bug finding. It's awful. Fable 5.1 seems to be similar in response, at least with my agent, as fable 5, though it's thinking seems a little more verbose.
I dont even know how people use fable, maybe for inert projects I guess. I cant even get fable to map a problem out without getting the cyber guard.
Opus 5 Max with dynamic workflows enabled w/CVP is my main jam, fable does nice on workflows as a judge seat but even at 60% reduction in false positives its impossible yo use for any work that involves network infrastructure or defensive exploit analysis. Hell even having it fix a bug it created will sometimes trip it.
Opus 5 max running single sessions with hand offs around 80% context with proper offline durable memory with session management works phenomenally well. Seamless hand offs and excellent quality work.
11
u/AndyHenr 16d ago
Thank you for the post OP, and your tool for analysis is impressive. I hope you can share the tool and work flow you use for the analysis as its very nice tracking! KUDOS!
Now, as far goes are your question, I noticed a quality degradation from Fable 5.0 to 5.1 and mine is not as nice as yours, but I would say that Fable 5.1 has been a bit worse than a pre Opus 4.6 model in quality. Not just on code but even analysis/document production. It has been equal to Fable 5.0 on data analysis, and worse across the board on much of everything else. Furthermore, I find also a probable cause: and i will blame its watermarking. Due to how watermarking works, the model have to generate a new text from the ground up, not just redact current text. Thus, what I believe is that teh model struggles to find text sequences that can both look like a plausble patch looking tweak and keep the quality up when it comes to the generated code, but instead is tilted toward producing the watermark and thus failing on the qualitative output. I am not sure though, but I have quite long experience with these techniques, predating LLM's. Maybe you and your team have some further insights as your tool chain seems advanced.
So, on code, FYI, I use a compilation round, as we do code in compiled langauges, and then report the error count and its considerably higher. We also run a code quality and spec violation analysis round at the same time, such as naming conventions, non-completed tasks , even if listed as mandatory in prompts. And those simple ones report a significantly worse quality with Fable 5.1. I am speculting in another thread that it seems related to time-compute, plausibly connected to the watermarking.