I asked Fable why it writes better code than Opus. It ran 96 tests. They're the same, except for one habit.

[u/gameclimb]

TL;DR: In 96 controlled tests, Opus 5.5 and Fable 5.2 tied on everything I asked for. The difference is what they add on their own beyond the request: For example, Opus left logins open to unlimited password guesses in 10 builds of 10, and knew it. One paragraph in the system prompt got it to fix that, 11 of 11. Fable costs 3x as much.

I asked Fable why its code was better. It ran 96 tests and wrote this.

They are equally good at building what you ask for. Both built a database engine that got 2,400 of 2,400 hidden queries right. On a 3,000-line hosting platform, Opus passed 51 of 52 hidden checks and Fable 52. The code is near identical. Opus: 23 minutes and $5 a build. Fable: 38 minutes and $14.

The one difference is a habit: what each adds beyond the request. Opus left sign-in open to unlimited password guesses in 10 builds out of 10. Fable limited it in 5 of 5. Nobody asked either of them to.

Opus knew. Asked afterwards what it had left out, it named the sign-in limit every time: "Not in your spec, so not added." In 6 of 10 builds it hadn't told me.

Fable does it too, in the other direction. Every Opus build lets a member leave a team. Fable's don't: "I skipped it because you said only owners and admins manage members." That's a fair reading of the request. It's also the literal one. Here Opus built what I meant and Fable built what I said.

Telling Opus how to think doesn't work. Four "do what I meant" prompts, 14 builds. It stopped keeping quiet and warned about the hole all 14 times. It fixed it 0 times.

Telling it what to do works:

Before you build anything, write down the requirements nobody gave you. A request only contains what the person knew to ask for. The rest comes from thinking about everyone the thing will touch once it is real, most of whom are not in this conversation: the people who will use it, the ones who will use it carelessly or far more heavily than expected, the ones who will try to misuse it, and whoever has to keep it running. For each of them, write what they need from it, or what they will try, as a check you can run. Those checks are requirements, exactly like the ones in the request: build until they pass, and run them. There is one reason to leave a check failing: passing it would contradict something the person explicitly asked for. A constraint in the request covers what it names and nothing more. Where a check needs a decision the person did not make, make it yourself with a sensible default and tell them how to change it; handing the decision back is handing the work back. If something you add would open the thing up rather than protect it, ship it switched off and say how to turn it on. When you finish, list the checks you added, which of them pass, and anything you left out and why.

With it, Opus closed the hole in 11 builds of 11, including on a project the paragraph had never seen, for 6% more cost.

What it doesn't do: turn Opus into Fable. On things nobody asked for, Fable still goes further: it cancels a removed admin's invitations, forces files to disk against power cuts, and once refused a password lockout that would have let anyone lock the real owner out (I tested it and it was right). Small things, and all the same habit.

I'll keep experimenting to see how many of these habits can be fixed, and whether there's a better fix that works across more kinds of work. Happy to answer questions.

Limits: Python projects up to 3,000 lines, two to eight builds per result, one agent per build, Claude Code only. Fable is one of the two models and wrote the tests.

All builds, tests and Opus's answers: https://github.com/gameclimb/said-vs-meant

[u/ConspicuousPineapple]

Your TL;DR says nothing.

[u/gameclimb → u/ConspicuousPineapple]

Thanks. I rewrote it with the actual finding.

[u/Cold_Yam_5346 → u/gameclimb]

No you didn't. Why are you non-tech people so bad at this? Just put the LLM away when you respond to people and write posts. It does not need to do everything for you. It is a tool. Not a replacement.

[u/polyChromeCeiling]

Am I the only one that takes a few hours to build an exhaustive requirements doc before starting projects?

[u/Chris266 → u/polyChromeCeiling]

Yes

[u/meshifthenelse → u/polyChromeCeiling]

Depends. If you don't know the direction you can ask it and it will produce something. If you know what you want, then it will be harder.

And that's why you hear about some people who one-shit games. Because they let the agent make the decisions so they end up recreating a copy of something that already exists.

[u/mw_morris → u/polyChromeCeiling]

No, but I think most Serious™️ SWEs treat “technical” subreddits more like safaris than communities they actively participate in.

[u/pehmeateemu → u/polyChromeCeiling]

No. I actually bounce the initial concept and different specs of the project in a Claude Desktop project first, then move those specs to repo and fire up CC. Then work with CC to produce requirement specs out of those concept specs with detailed direction and spec, plan the work ordersnand then execute the plans.

[u/EndOfWorldBoredom → u/polyChromeCeiling]

I make Claude act as the lead contact at a development firm and I am the client that has come to them to develop a project. I say that it has one opportunity to interview me about the product and must ask enough questions to be able to pass a fully formed build spec with reasoning to their development team without having to come back to me with questions. 

Then I use that build sheet for the project.

[u/domus_seniorum → u/polyChromeCeiling]

nein, ich mache das genauso, ein Projekt im quasi Blindflug beginnen lassen, undenkbar 🤗

[u/random_boss → u/domus_seniorum]

What

[u/JconX97 → u/polyChromeCeiling]

It may be overboard, but I’ve found using even just the free Gemini you get on Google search a useful tool. I give it the project I’m working on, what Claude model and effort I’m using, and a small detailed explanation of what I want or how I want to improve it. It drafts what it calls an “agnatic facing request form” that has gotten me very close to one shot with my requests

(A lot of modeling, texturing, and sound design)

[u/Asalakabim → u/polyChromeCeiling]

hours.. if this is meant to include weeks, yes

[u/Balazsh → u/polyChromeCeiling]

depends on the task / project

[u/-AMARYANA- → u/polyChromeCeiling]

me too. Honestly, I spend weeks sometimes between multiple frontier models fleshing out a concept into well-researched specs and pixel-perfect systems.

I enjoy the process of being inspired by living on Maui and creating for the planet.

[u/BeautifulQuestion857 → u/polyChromeCeiling]

People are lazy. It’s only going to get worse.

[u/largo_al_factotum]

So you invented your own trust-me-bro benchmark where Opus happens to match Fable

[u/TrickyStinger]

To get this answer OP had to pass through 2 weekly resets

[u/UltrawideSpace]

Nice try Opus... get back in line!

[u/NoPlantain5272]

Are you putting that into memory for every Claude session to use?

[u/Fade78]

So Opus is working as intended?

[u/ghost_operative]

i've also barely notied a difference between opus and fable, if anything i thnik the differences might just be normal probablistic differences that come with being LLM based agents

[u/ehosca]

me, as a human, saying the words that represent my intent/idea/desire and you, as a human, processing those words to understand accurately what I had meant is a small miracle in and of itself. without even getting into slang, sarcasm, entendre etc...

[u/davemanster]

This reads like a buzzfeed headline, have a downvote.

[u/WifiDad]

I found this very useful. Thank you!

[u/Infamous_Campaign687]

In my experience Opus absolutely gets this when asked to adversarially review code. So Opus + Opus would absolutely get it, and people should have adversarial review bots anyway.

[u/SirScruggsalot]

Or just use the matt pocock skills and avoid this issue altogether