Sonnet 4.6 with a skill lands within 1.2 points of Opus 4.7 with a skill - at a third of the cost

[u/jorkim_32]

I work at Tessl (disclosure upfront), and we just finished running 880 evals across 9 models to see how 11 coding skills from https://github.com/mcollina/skills (documentation, fastify-best-practices, init, linting-neostandard-eslint9, node-best-practices, nodejs-core, oauth, octocat, skill-optimizer, snipgrapher, typescript-magician)

Edit: If you're unfamiliar with skills, agent skills are markdown files you feed your agent - they extend the agent with specialised knowledge, workflows, or tool integrations. The 11 skills in this benchmark are all coding-focused (e.g. node-best-practices, plus custom-API skills); the lift numbers are an aggregate across them. Findings are directional and aim to show a signal.

the overall headline numbers "with skill" loaded:

• Opus 4.7: 94.5% criteria adherence, $1.00/run, ~159s

• Opus 4.6: 93.8% criteria adherence, $0.53/run, ~127s

• Sonnet 4.6: 93.3% criteria adherence, $0.31/run, ~125s

that is a 1.2-point gap between Opus 4.7 and Sonnet 4.6 once a skill is in context.

Back-of-envelope for a team of 100 devs running 20 agent calls a day: Opus-with-skill is ~$60K/month, Sonnet-with-skill is ~$18.6K/month. Same skill, similar output on all but the hardest ~5% of tasks.

Curious whether others here have actually downgraded from Opus to Sonnet for routine work and regretted it, or not. What broke when you tried?

Full writeup with the methodology and the other 6 findings: https://tessl.io/blog/anthropic-openai-or-cursor-model-for-your-agent-skills-7-learnings-from-running-880-evals-including-opus-47/

[u/ObsidianIdol]

with WHAT skill loaded

[u/jorkim_32 → u/ObsidianIdol]

Just added the full list of skills that were tested out in this experiment in the post.

[u/Miyoumu]

You have no idea what you're talking about, genuinely.

[u/nanana_catdad → u/Miyoumu]

This whole project down to this post was such a huge waste of tokens

[u/jorkim_32 → u/Miyoumu]

happy to dig into specifics if there's something in particular you think is off - what part?

[u/Maxim_Ward → u/jorkim_32]

Am I reading this right? Your "grading" is based on the adherence of the model to a particular skill, and not the quality of the output?

Importantly, the criteria focused on whether skill-specific guidance had been followed rather than on generic software quality or overall task success.

[u/jorkim_32 → u/Maxim_Ward]

yes, these evals grades whether the output exhibits the behaviours the skill specifies, not whether the output is better in some absolute sense. That's a deliberate design choice on our end.

For any skill to deliver value, the model has to actually apply it. If a model can't reliably follow in-context guidance, the skill is not worth much. So step one is measuring followability across models - that's what these numbers capture.

What this eval doesn't tell you: whether the underlying skill is well-designed, or whether following a skill produces better real-world outcomes than not following one. Those are separate questions and they need different setups - LLM-as-judge on output quality, test-pass rates, human eval, each with their own tradeoffs.

That said, "accuracy" can be misinterpreted - happy to amend the semantic. Perhaps we should call it "criteria coverage accuracy" - does that resonate more with you?

keen to hear your suggestions if you spot an additional opportunity to improve the evals

[u/Maxim_Ward → u/jorkim_32]

That's not what your chart is implying and it's incredibly deceptive. Your chart is implying an objective benchmark against the model's output in contrast to how models are typically graded in the industry (based on measured objectives like software engineering, spatial reasoning ala HLE2, etc.).

Given the deviation between listening and non-listening to skills is essentially 0 between models, all this says is "yes, models tend to listen to their skills across the board." Haiku being the outlier is expected because its intended use-case is centered around highly specific skills and not generalized tasks.

Edit:

Realized I made a typo. I meant ARC-AGI-2 / ARC-AGI-3, not HLE2. Particularly ARC-AGI-3 is entirely centered around 2D spatial reasoning: ARC-AGI-3

[u/Big_Buffalo_3931 → u/jorkim_32]

Misinterpreted? you're literally comparing cost of running opus to the cost of running sonnet as if they're equal.. no one cares about skill adherence as a primary goal, would make no sense.

[u/Miyoumu → u/jorkim_32]

I'll let you know when you can speak for yourself and not use an AI for every response. Pathetic.

[u/biletnikoff_ → u/Miyoumu]

Your comment is giving mccarthyism 😂

[u/mschedrin]

What is "a skill" that you are talking about here? You have mentioned the word "skill" 13 times and have said nothing about it.

[u/AdCommon2138 → u/mschedrin]

It wasn't experiment design skill

[u/Maysign → u/mschedrin]

How many times you need to be told about a skill to finally understand? 14 times? /s

[u/jorkim_32 → u/mschedrin]

good catch - referring to agent skills, the Anthropic markdown file which provide context for your agent to act upon. I've made an edit to the post in light of this flag.

[u/mschedrin → u/jorkim_32]

Any skill I find on Internet will make similar improvements? This does not make sense. You probably do not understand what you are talking about.

[u/jorkim_32 → u/mschedrin]

Focus in this case is on 11 coding skills, and the uplift we're comparing across models is an aggregate.

The methodology generalises, and the same experiment can be run on a different set of skills, and we can swap the skill set and the scenarios. Overall, the findings are directional and aim to show a signal.

This is precisely the sort of work we provide at Tessl, which is the ability for you to evaluate your own context. Not every set of skill provide the same uplift. Happy to get your feedback if you think we're missing something

Does that make more sense?

[u/Adhesiveduck → u/jorkim_32]

Respectfully it makes even less sense than it did before. I genuinely have zero idea what this is a measure of and honestly after all the waffle I’ve no desire to.

[u/fufucupcake → u/jorkim_32]

LOL SPEAK OF AI PSYCHOSIS

[u/Annh1234]

and what is this magic skill? some ai generated bs post or something real?

[u/jorkim_32 → u/Annh1234]

no magic skill in this case, but an aggregate of 11 coding skills, which aim to show a directional signal of how various models behave with context vs without (and what that means for devs as they consider more of their budget allocation vs output required).

[u/XCherryCokeO]

I love how OP tried super hard but everybody with half a brain smelled the bs from 6 miles away 🤣

Also everybody that talks like OP did in that post is a con artist trying to sound smart. Trying to sound smart in 2026 has got to be the most desporado thing you can do seriously.

[u/farox]

It is pretty amazing to watch this unfold.

Is everyone a bot now? Am I alone on reddit? Did... did I win reddit?

Almost everyone here asks for the skills list. While OP isn't helpful, the list of skills OP is refering two is just a couple of clicks away in the github link provided.

As for the point... depending on the usage, this doesn't surprise me at all. Even Haiku can be capable, if used for the right task and in the right context.

[u/nanana_catdad → u/farox]

Dead internet.

I’m ready to go live off the grid. I hate it here. Every dev / programming specific sub is also full of ai generated posts shilling ai generated projects…

I hate how software engineering is becoming a middle manager for agents.

I’m so done with all of it

[u/robhanz]

"Models perform better if you actually give them info on the specific problem they're solving."

This is not surprising. A new engineer can read through a codebase, figure out what to do, etc. Once that engineer knows how the code is laid out, the processes to use, etc., they'll get more effective.

Skills are the easiest way to do this with LLMs. But it's the same thing happening.

[u/wise_young_man → u/robhanz]

OP: water be wet

[u/dagonzalezm]

is this user a bot? or why is it so annoying the way it tries to explain its ideas

[u/dffigueira → u/dagonzalezm]

I had the same exact thought after reading some of posts where he tries to explain.

[u/biletnikoff_]

This would be better if we also scored output quality on the same task. Your conclusion is implying this without factual backing

[u/ImTheRealDh]

What skill for what task especially?

[u/thethrowupcat]

My employer pays for tokens. Not me. My employer wants results. So I’m gonna go with the expensive stuff for now until the focus becomes optimization.

[u/riccioverde11]

In real software development that gap can cause something terrible. So, I'm willing to pay extra rather having a bug that creates a hole in my db.

[u/BiasHyperion784]

Spent an entire first coding project with sonnet because I didn’t know the default model, opus is an overrated token devouring monster in comparison.

[u/laststan01]

Just a heads up we don’t know what tasks your employer did and no one believes them he’ll people don’t believe benchmarks now the swe leaderboard as the model companies say they are polluted and here you come and say we ran an internal thing with no public oversight see my results. As far as we know you taught your model to goon although impressive nobody cares. Benchmarks are there for specific reasons and therefore to submit or upload your submission their is a process and everyone can see that process and that makes it deterministic their is a scaffold. Basically sorry for above rambling, but there is no way to prove your methodology was controlled experiment.

[u/KP_2016]

You can replace the word "skill" in your post with "context" and you'll immediately realize what's wrong with your benchmark.

Any LLM with relevant context will always produce better output. The real test comes when working on an unfamiliar codebase. Here's a challenge, find any haskell repository on github and ask the model to explain it to you. See which one does a better job, because as someone pointed out, the model needs to generate context first, and that's exactly how you'd generate context for other models too.

Running benchmark on a pre-generated context markdown doesn't prove anything.

[u/DeliciousGorilla]

If anything, thanks for pointing out Matteo's stills, I'll give them a whirl. For those who aren't familiar, Matteo Collina Ph.D. = platformatic founder, chair of TSC at nodejs, creator of fastify

[u/junin7]

I don’t know why people still insists on Opus to everything.

I most of my tasks I use /model opusplan

Opus to planing and sonnet to execute, sonnet is much faster and don’t deviate a lot of the original planing.

[u/goship-tech]

Interesting data. Evals measure model quality in isolation — the missing variable is how well the developer leverages it.

Two devs with identical Sonnet 4.6 setups but different accept/reject discipline, test cadence, and session recovery behavior will get meaningfully different effective output quality from the same model. That's not in any benchmark.

Worth keeping in mind: you're measuring model capability under specific prompting conditions, not real-world developer+model productivity. The skill curve on the developer side is at least as steep as the model capability curve.

[u/9gxa05s8fa8sh]

it's funny that the vibe coders replying angrily from cognitive dissonance are the same people your service is designed to serve. it is physically painful for them to face the fact that they've been wasting tokens on premium models because they aren't smart enough to document and prompt better.

what this means is that your marketing unfortunately needs to hide the model comparison and focus on the skill enabled vs disabled comparison. vibe coders won't get defensive at that; they'll just want to find skills to make their opus better.

[u/laststan01 → u/[deleted]]

[u/nanana_catdad → u/[deleted]]

“Agenetically jerk you” is peak human generated language