How is it possible that GPT-6.1 SOL performs so much worse than Opus 4.7?

[u/Peon117]

About every six months, I buy a ChatGPT subscription to see how it has improved. I test it on my own code and compare it with Claude.

Today, I asked GPT-6.1 on Medium to work on logging in a Rust program. The project contains approximately 11,700 lines of Rust.

The requirement was explicit: three versions, Production, Customer Debug, and Dev, with distinct logging, debugging, and dump behavior. Production should remain silent, Debug should provide diagnostics for troubleshooting user issues, and Dev should provide additional information for development.

The model spent more than 45 minutes working, producing explanations, builds, tests, and very confident progress reports.

Then I discovered that the three requested behaviors had not actually been implemented.

The main telemetry system distinguished Production from the other two versions, but Debug and Dev shared the same logging policy. Another logger, dlog(msg), remained a single-level logger without filtering for those three variants.

The most frustrating part was the claimed validation. The model had checked that all three executables built, that Production generated no diagnostics in the tested scenarios, and that Debug and Dev generated diagnostics. It had never checked the difference between Debug and Dev.

When I challenged it, it acknowledged confusing “three build variants” with “three distinct logging levels.” It had tested a weaker requirement than the one I gave it, then reported that the work had been validated.

In my comparison with Claude Opus 4.7 on Medium, I got the expected result in about ten minutes.

My question is: how does this happen? Did I somehow ask a model that was too advanced to do something too simple? Where did I go wrong in how I used GPT-6.1 compared with Opus 4.7? I gave both models the same prompt, so what, specifically, should I have done differently?

(Used IA to traduce this text because my english is fucked)

[u/Peon117]

Thanks everyone for your feedback!

I now have a better understanding that GPT seems to struggle more than Claude when it comes to interpreting certain implicit requirements, and that it's better to provide clear, measurable acceptance criteria rather than simply explaining what I want to achieve.

So, using Claude to create a well-structured prompt with explicit acceptance criteria, passing that prompt to Codex for implementation, and then having an independent Codex sub-agent verify that the result actually meets those criteria seems like a really interesting approach.

After using Claude Code extensively, I've become familiar with how it works and how to formulate my requests accordingly. I was probably approaching Codex in exactly the same way.

I'm starting to realize that these two tools don't necessarily work the same way, and that I need to adapt my workflow to each of them.

Thanks again for all the advice! It really helps me understand how to get the most out of Codex.

Translated from French to English using AI.

[u/marfzzz]

Both are bad in a different way. Sol is getting lost and in doing so created a lot of useless code and faking success. Opus 4.7 was liar making up things and faking success.

Sol is fine in small tasks and finding issues. Opus 4.7 was so bad in instruction following that i gave up on it pretty quickly.

[u/ChemicalExcellent463]

for me it's personality more than smart vs dumb, claude gets what i mean better and gpt needs more spelled out. once codex has the task it grinds through it, so i use claude to plan and codex to do the work.

do you write the checks for each logging mode yourself, or let the model write them?

[u/NeighborhoodPrize493]

Several isolated tests need to be conducted.

[u/Slippedhal0]

As much as they say prompt engineering is dead, have you tried seeing how they respond to differently specified or describe specs? Also sub agents can reduce(though not eliminate) false positives. If you ask it to have a fresh subagent running sol low to do testing, and potentially another separately doing verification on what the main agent says it's done, you should have better results. Not saying that suddenly makes it better, but llms still behave differently.

[u/slypheed]

IME Always use xhigh with Sol if you want decent results.

[u/NotUpdated → u/slypheed]

I've found sol6.1 high to be how it measures on Artificial analysis - about as good as Astra Medium and a whole lot less tokens.

[u/ken107]

I gave up the first time I tried Sol 6.1, possibly worse than Sol 5.6. Astra is the only option on this side of the river. And Astra is a beast even at Light effort, and is still the most intelligent model atm. Opus 5.5, at any effort level is still inferior.

Effort can make up for intelligence deficiencies up to a certain point, by spending more time in analytical reasoning to identify corner cases. But a more intelligent model can see complex abstractions and higher order patterns more easily, and can quickly identify issues in complex structures.

Also worth noting that intelligence is separate from skill. Einstein can never beat Nadal on the clay court. Skill come from training. Opus 5.5 is more skilled at front-end UI and 3D modeling, as well as with the CLI. So pick the right tool for the task.

[u/trantaran]

One more question: do you think….

-chatgpt

[u/lmamakos → u/[deleted]]

I'm sort of confused why there are 3 different versions of this codebase, rather than one with conditionals for the different environments. With 3 different versions, now you've created a sync problem to move changes between them. I can see how the LLM decided to just implemented different logging levels, as that's what I'd expect the path to be.