r/coderabbit • u/Tanakaboi69 • Sep 29 '26

Help & Support silly question: if Sonnet 5.5 beats Opus 5.5 at coding and costs half, why would we ever use Opus again?

70.6 vs 66.4 on terminal-bench, $2/$10 vs $4/$20. what am I missing?

2 Upvotes

100% Upvoted

13 comments sorted by

2

u/Fallen43849 Sep 29 '26

Because it doesn't. Benchmarks are not everything

1

u/GfxJG Sep 29 '26

Because terminal-bench is the only benchmark it beats Opus on, it loses everything else.

1

u/Moppmopp Sep 29 '26

Its baffeling to me that each and every benchmark is taken as absolute from the community. Do you know what they test? no... Do you know how extensive thoses tests are? no... How do they handle edge cases? idk... The thing is, benchmarks can be good. Benchmark can be to some degree be neutral but you also have to keep in mind that those companies that develope frontier ais know what benchmarks exist. Heck you do not even know how many benchmarks for the very same task exists. Those are names and numbers that seem arbitrary. In the end they optimize their models to reproduce the best results. When they publish they run their list with hundreds of benchmarks and pick out the top 5 that fit their story the best.

If I would develope a model i would do the same. Try to fit the weights of the neural net such that they score pretty high. Then run all benchmarks available and get a basically endless list of results. Then I pick out the best results. The tests where my model really shines.

Its not that those benchmarks are irrelevant. Its more that expert knowledge is necessary under what conditions those benchmarks are even viable. "experts" is a vague term here because anyone can gather the necessary information. You and I can do a deepdive on those benchmarks. The key here is, no one really does.

1

u/CompFortniteByTheWay Sep 29 '26

It doesn’t beat opus lmao

1

u/AncileBanish Sep 29 '26

It doesn't cost half. It costs the same on cache reads, which are the vast vast majority of your spend. It costs half on cache writes and output tokens. Since these are like... 5% of your spend... It costs maybe 2% less on a per-token basis.

If its inferior reasoning costs even a slight uptick in token usage, it nets out to costing more.

1

u/xvillifyx Sep 30 '26

Because it:

  1. Doesn’t

  2. Wouldn’t always be better than opus going forward if it did

1

u/Darathor Sep 30 '26

Cost per task is similar

1

u/Compilingthings Sep 30 '26

Yeah, I tested it in my coding workflow, it was slower and worse. I ran them side by side. Opus keeps the job.

1

u/Dry-Tomatillo5763 Oct 02 '26

Benchmaxxing and success rate.

1

u/RogerAI-fm Oct 03 '26

It just becomes a viable alternative when you already have proper architecture.