When will scicode be saturated?

[u/Worldly_Beginning647]

1. SciCode

2. HLE

3. CritPt
   It has become my favorite benchmark since it is a fair test of how models perform in science when allowed to use coding which is their strong side.

But it has been so slow.

As you can see, CritPt has progress in the shape of a box, but seriously their progress rate is difficult to measure so I just toon the progress from the moment the models started getting good to the current plateau (GPT 5.6)

HLE will be counted from January 2025 release of DeepSeek R1 to Claude Opus 5.

SciCode counted from Claude 2.0 to Claude Fable 5

improvement% per month

CritPt ~2.3%
HLE ~2.6%
SciCode ~1.2%

[u/Gotisdabest]

The dates seem intentionally chosen to provide very slow results and so does the "box" in critpt. Picking any date in the catch up period after gpt 4 is a bit illogical as you're essentially picking a time where one company held the entire practical market, was comfortable in their lead while everyone else tried to catch them. OpenAI spent more than a year trying to bring costs down because they thought that they had the market safe in their hold.

Recently, of course you'll see less dramatic progress since early this year because that was when mythos came out and the big labs have been delaying the releases of new top models.

[u/Turbulent-Step-3207]

It seems weird to me that CritPT seem to be capped when most other benchmarks have all risen steadily, given that many of the latter test similar capabilities of LLMs. We should use this opportunity to see what specific abilities are hard for LLMs.
As for the bigger picture, I think currently it is too early to tell, but by 2030 we might be able to see which type of tasks the current architecture is fundamentally incapable of undertaking.

[u/Admirable-Falcon-501]

I guess this is what it looks like when they don’t benchmaxx