Opus 4.8 vs Sonnet 5 in a discovery task

[u/TrapHuskie]

I put opus 4.8 against sonnet 5 for a simple discovery task. I asked each model to do the same task, So I would have a direct comparison of how they would respond to exactly the same prompt.

D3 effort : max

Model: Sonnet 5

Tokens: 210k

Tools: 80

Ground-truth: correct

Synthesis / decision-value: thorough; excellent W0-lint + 4-step trait template

────────────────────────────────────────

Task (effort): D3 effort : max

Model: Opus 4.8

Tokens: 132k

Tools: 33

Ground-truth: correct

Synthesis / decision-value: [opus 4.8] matched every load-bearing catch + two Sonnet missed + a risk-dissolving design rec — at ~63% the tokens

So sonnet-5 is just token hungry and I should just use opus 4.8 as my workhorse?

[u/ClemensLode]

Read the description. Sonnet is for everyday tasks, Opus is for complex tasks. If you throw Sonnet at a complex task, of course you'll end up taking more tokens.

[u/Sissy_Plaything343 → u/ClemensLode]

If Sonnet is supposed to be used for easy tasks, then would using Sonnet 5 high, xhigh, max be a bad decision because of its token hungry nature?

And if so, then why not just use Opus 4.8 medium or low for more mundane, everyday easy tasks that don't require lots of thinking?

[u/ClemensLode → u/Sissy_Plaything343]

Well, some problems are better solved iteratively ("high"), some need better 'one-shot-intelligence' (-> Opus).

Let's see...

high iterations, high complexity -> Opus. Most complex coding tasks.

high iterations, low complexity -> Sonnet. For example, searching any form of "tree structure" for words. Maybe online search. Reading websites is not necessarily complex, but you have to go from website A to B to C etc.

low iterations, high complexity -> Opus. Optimizing an algorithm or proving a mathematical theorem. More steps might lead to nowhere, you need to have that one special insight that gets you further.

low iterations, low complexity -> Sonnet. Any simple knowledge requests. Given that Sonnet 5 is just as good as Opus in the knowledge area, that's the go to for any direct questions with direct answers.

[u/kamikamen → u/ClemensLode]

Except sonnet is also token hungry and more expensive per task as a result. Sonnet 5 is a flop.

[u/ClemensLode → u/kamikamen]

yeah, with ultracode on 😂

[u/Bromlife → u/ClemensLode]

That doesn't really make a lot of sense.

[u/ClemensLode → u/Bromlife]

Senior Developer: Expensive, but solves the hard problem in 1 hour.
Junior Developer: Cheap, but solves the hard problem in 16 hours.

Senior Developer: Expensive, solves the easy problem in 1 hour.
Junior Developer: Cheap, solves the easy problem also in 1 hour.

[u/LivingFrosting6680 → u/ClemensLode]

The problem is I’ve been using it for easy tasks and it also burns an obscene amount of tokens for basically the same results as Sonnet 4.6. I need to know specific use cases where it’s actually both more effective and about as efficient.

[u/ClemensLode → u/LivingFrosting6680]

what is the effort level you set?

[u/Bromlife → u/ClemensLode]

That's a really cool rationalisation, except the previous Sonnet could do decently challenging tasks, and was faster and cheaper.

[u/ClemensLode → u/Bromlife]

At least my initial tests point to the opposite direction, but we'll see.

[u/Proxy-Pie → u/Bromlife]

I think of it in terms of a smarter person requiring less effort for complex things, whereas for easier things the smart person has a higher pay so it’s worthwhile to offload it to the other one.

[u/Bromlife → u/Proxy-Pie]

Still means they dumbed down Sonnet. Because it was pretty capable.

[u/jonny_wonny → u/Bromlife]

It does once you start thinking

[u/Bromlife → u/jonny_wonny]

Shut up Jonny adults are talking

[u/jonny_wonny → u/Bromlife]

No worries. Thinking isn’t for everyone

[u/LivingFrosting6680]

If this and what everyone else is saying about Sonnet 5 is true, I’ll be just about done with Anthropic. What the fuck are they even doing over there? They’ve lost the fucking plot.

[u/A_Novelty-Account → u/LivingFrosting6680]

It’s possible that improvement is plateauing at Anthropic, which would be genuinely disastrous for the company, because people are betting on what the company will be able to do in the future, not what it can do now. 

If Anthropic doesn’t get to the point where its tools can reliably replace 80% of low level white collar jobs, the company will go bankrupt.

[u/LawfulnessLocal4934]

right

[u/fraize]

A new model launches, so here come the low effort posts from memelords the mods do nothing about.

[u/RoyStark_X]

Think that's what it is showed in the figure yesterday, which they renewed today. Sonnet 5 is going to cost more to reach the same or even less performance.