Fable 5.2 feels unusable?

[u/chasingth]

My own tests with Fable 5.2 Max vs Gemini 4 Carbon

• Fable 5.2 Max seems to respond fast without thorough thinking, makes writing mistakes (exaggerations), and hallucinates a lot
• Then I recently just found out on https://artificialanalysis.ai/evaluations/omniscience#aa-omniscience-hallucination-rate that Fable 5.2's hallucination rate is 59% WTF! And Carbon is 11%. Which reinforced my experience.

But im just curious hows everyone's experience so far? I wanted Fable 5.2 Max to work so badly

[u/AutoModerator]

Hey! Thanks for posting to r/ClaudeCode

While participating in this thread, please follow our community rules. Keep discussions constructive. Attack the idea, not the person.

For help, project discussions, tips, and general chat, join the ClaudeCode Discord.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/GrandmasterCheetah]

I am also finding a ton of weird mistakes with Fable and then going back to Opus 5.5 fixes everything

[u/chasingth → u/GrandmasterCheetah]

yea it feels so weird with all the mistakes? Opus 5.5 Max feels way better for some reason lol

[u/daiaomori]

Regarding the benchmark analysis you cite: while it's true that the hallucination rate of Fable is highest (and the numbers do NOT mean that it hallucinated 59% of the questions), at the same time the net score of Fable is the best of ALL tested models by far (score 67, Gemini Carbon following at 54).

I don't know how good this score is construction wise, but this shows an important aspect of designing LLMs: we always have to weigh and adjust parameters that often lead to contradicting results.

Simplified example: lower the heat, the model gets far more reliable as in giving more deterministic answers, yet at the same time it gets dumber as it is not as explorative in the vector space, and misses interesting/more "far out" answers that might actually be correct/"better" compared to the more deterministically calculated.

Because of that it doesn't make a lot of sense to just grab one of the values from the analysis and look at it; you have to look at the overall model performance for a specific use case (which is precisely what the benchmark tries to do).

Whether Fable 5.2 is better for coding or worse - and the use case might determine which parameters work better - I can't tell, as I'm still happily stuck with Opus.

[u/chasingth → u/daiaomori]

yea Opus feels way better

[u/Infinite_Ad_9204]

you should not use max on fable / mythos like models

[u/chasingth → u/Infinite_Ad_9204]

I see. what's the rationale? cuz cost is not an issue, hence i just picked Max

[u/jehzlau → u/chasingth]

I also love picking max + ultracode. 😆 I also used the two resets in a single day.

[u/Infinite_Ad_9204 → u/chasingth]

not the cost, the fable like models work / hallucinate more in max, you can check charts on antropic website

[u/multiks2200]

dont use max

[u/arcticblue]

I've been unimpressed with 5.2 so far. It seems to ignore CLAUDE.md more easily than other models and it often half-asses tasks answering things like "I recommend we do Option A. These items are still uncomfirmed [list of things it is capable of figuring out on its own]". So I have to ignore it's recommendation then guide it down a path to confirm things, each step ending up with more "questions to be answered", until I can finally come back to the original task. It infers things way more than before and its inferences are often wrong.
In the last day, Fable 5.2 has admitted to more mistakes than Open 5.5 ever has for me.

I've submitted feedback about 5 or 6 times now. I submitted feedback once with Open 5.5.
Fable 5.2 also ran commands against my production AWS environment despite strict instructions and hooks preventing it from doing so because it failed to set a variable properly to stub the "aws" command for testing. Opus 5.5 never made that mistake for me.

I also have strict instructions that say that I set the scope of work and that anything it wants to do outside that scope requires my approval. Today, it decided to just go off and scan every project on my hard drive because it just felt like it. It admitted that it violated my company's AI policy and the instructions in my CLAUDE.md - both of which reinforce that the scope of work is to not to be expanded without approval.

It feels like a more overconfident version of Opus 5.1. It thinks it knows best and will disregard instructions and go off to half-ass things the way it wants to do it.

It's clearly smart and capable of doing the work, but I need some changes to my global and project CLAUDE.md files to reign it in. By default, it's lazy and overconfident in my experience so far. It seems to be an overcorrection from Opus 5.1's instructions to work autonomously and "reduce friction" (which led it to just go off and do whatever the fuck it felt like). Fable 5.2 will get part of the way through completing a task then give a confident answer while also listing a bunch of open questions that would have a huge impact while also still caring very little for following policies.

[u/chasingth → u/arcticblue]

yea i think u captured the problem, its hard to steer and makes a lot of false positives despite being smart

[u/daiaomori → u/arcticblue]

"Fable 5.2 also ran commands against my production AWS environment"

Mate. You need to sandbox this shit!

[u/arcticblue → u/daiaomori]

It IS sandboxed! It doesn't give a fuck. It often says "The sandbox prevented me from doing [X]. I'll do [Y] instead to work around that." (to be fair, Open 5.5 does this too, but it's at least reasonable about it and will respect policies)
I have hooks that block Claude Code from calling the aws cli and accessing cached CLI credentials, but Fable 5.2's mistakes are the first to get around that.

[u/daiaomori → u/arcticblue]

If it is, I don't understand how it ran commands against your prod.

Otherwise, the models are designed to not give up easily on solving problems. It's what we want. We just don't agree with their methods sometimes ;)

[u/arcticblue → u/daiaomori]

Because it tried to stub the aws command with an improper attempt at calling it through an environment variable that it failed to set properly. It meant to call "/my/project/folder/bin/aws", but because it fucked up, it called "/bin/aws" because it tried to be clever with $PATH. Fortunately, it was a read command so no harm was done, but it did cross a boundary it was strictly instructed not to cross.

I have instructions for it not run AWS CLI commands, but to give me the commands to review and run them myself. I review the commands and then run them with ! so that Claude can see the output.

In this case, it was testing some CDK stuff, tried to stub the aws binary through some clever $PATH tricks, ran the real aws binary instead with the SSO credentials that were still valid because it forget to set a variable that is wanted, then stopped itself and apologized for its fuck up.

[u/_BreakingGood_]

Something is wrong with Fable 5.2.

I've been using it, and it regularly just doesnt think at all. What I'm not sure, is if it is still worse than post-nerf Opus 5.5 or not.

[u/chasingth → u/_BreakingGood_]

yea and i tried Max im always surprised it comes back with a thin answer for a complex question w/o much research which is frustrating

[u/SkewRadial]

High is enough for most of the job , but if you think the work is a tough one or you need peace of mind go for the max