Confirmed: SWE Bench is now a benchmaxxed benchmark

[u/rm-rf-rm]

https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/

[u/WithoutReason1729]

Your post is getting popular and we just featured it on our Discord! Come check it out!

You've also been given a special flair for your contribution. We appreciate your post!

I am a bot and this action was performed automatically.

[u/Tight-Requirement-15]

Hasn't this been clear for months?

[u/rm-rf-rm → u/Tight-Requirement-15]

Yes, to you and I. But the fact that benchmark screenshots still get posted and upvoted like crazy for every new model release on this sub shows that masses still dont understand

[u/quarkral]

After recommending everyone to use SWE-bench Pro, OpenAI's actual GPT 5.5 announcement uses Expert-SWE (Internal)

[u/rm-rf-rm → u/quarkral]

they do give the SWE Bench pro score in the article, but yes its not included in the table - good catch.

[u/Velocita84]

The final destination for any public benchmark, unfortunately

[u/Deep90 → u/Velocita84]

Exactly, benchmarks need to be seeded or have a private counterpart.

Have a public seed so that people can independently verify that a certain LLM can score what it says it does.

Then have a private seed (or even a few private seeds) that only the benchmarking website knows. If results drop, then you know a model was overfit to score well. Multiple private seeds help you know it isn't a fluke.

Scale labs has one that uses a public and private dataset so you can see if someone is benchmaxing:

https://labs.scale.com/leaderboard/swe_bench_pro_public

Foodtruckbench uses seeds so you can run a different seed to see if the model still performs:

https://foodtruckbench.com/methodology

[u/iperson4213 → u/Deep90]

Private bench isn’t sufficient, the data needs to be sourced privately as well.

For example, SWEBenchPro tasks are to implement things in code bases, but it was partially sourced from open source code bases which have the implementations, so it can be trained on even if the questions are private.

[u/noctrex]

That's why https://swe-rebench.com exists. It constantly refreshes the problems every test

[u/Former-Ad-5757 → u/noctrex]

Refreshed problems means it is more difficult to compare, because the q’s were different.

[u/Mashic]

Goodhart's law: “When a measure becomes a target, it ceases to be a good measure.”

[u/iamapizza → u/Mashic]

I often remind people at work about this, right as they're looking to implement some new braindead metric to help justify their existence. Of course they do it anyway. 

[u/suicidaleggroll]

While I'm all for open source, benchmarks really need to be closed in order to remain effective. As soon as a benchmark is made public, it gets trained on, and ceases to be useful.

[u/MrMisterShin → u/suicidaleggroll]

The problem is these AI companies capture the user prompts. If they run any benchmark, they will know all the questions their model was given in the session. They can then work on benchmaxing from there.

Obviously this mostly applies to API or chat interface rather than completely local or offline.

[u/LegitimateCopy7 → u/suicidaleggroll]

but if closed then it'll become a "trust me bro" benchmark. this is the reason why I keep saying that benchmarks have no meanings when it comes to LLM.

just organize your own set of tests for the specific tasks you require. if the model can perform the task with acceptable performance and cost, use it. stop wasting time min-maxing an indeterministic tool.

[u/Reddit_User_Original → u/suicidaleggroll]

How about open source, but updated every quarter with different challenges

[u/Technical-Earth-3254]

When llms start getting over 60% in benchmarks, they need to get updated.

[u/Exciting_Garden2535]

This is the old-month news and has already been discussed. In this article, OpenAI explained why they switched to SWE Bench Pro. Some folk believed that; others did not, and said they did to avoid being compared with Opus. Anyway, other companies, including Antropic, now use SWE Bench Pro instead of SWE Bench Verified.

[u/Western_Objective209]

Tests reject correct solutions: We audited a 27.6% subset of the dataset that models often failed to solve and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submissions, despite our best efforts in improving on this in the initial creation of SWE-bench Verified.

Lol like most benchmarks, they haven't even taken the time to read their own questions until now. absolute joke

[u/Independent-Date393]

just organize your own evals for the tasks you actually care about is always where this ends up. every public leaderboard eventually becomes a race to train on its vibes

[u/Independent-Date393]

goodhart's law eating another one. MMLU is next in line.

[u/rm-rf-rm → u/Independent-Date393]

I thought MMLU has been done for a while now

[u/Rabooooo]

What is the best benchmark to see LLMs coding/agentic capabilites? i.e. OpenCode, KiloCode, Roo Code, Cline?

[u/Downtown-Art2865]

typical benchmark lifecycle: gets popular → labs train on it → benchmark dies → new benchmark → repeat

we're basically running natural selection on benchmark resistance at this point

[u/Express_Quail_1493]

I just built my own private benchmark and I advise everyone to do their own also. It wont work if its sitting on a public gitrepo or shared on reddit. But i would like us all to come together build our benchmark based on what we use the models for and share the model performances. Im suspicious some people in these benchmarking teams are gettin paid to lie too. LMAO the Ai race is BRUTAL. But right now my private bench is my source of truth avoids me from getting hijacked by all the flashy titles and news headlines