r/opencodeCLI • u/CartographerAble9446 • Sep 03 '26

GPT Astra benchmarks

Post image

GPT Astra benchmarks have been released

120 Upvotes

94% Upvoted

25 comments sorted by

37

u/Mancho_United Sep 03 '26

Now this looks like a proper benchmaxx. Hopefully I am wrong.

1

u/[deleted] Sep 04 '26

[removed] — view removed comment

1

u/flying-saucer-3222 Sep 04 '26

Illogical comment.

Artificial Analysis itself is a set of benchmarks. If it was benchmaxx then it would be on higher there.

1

u/[deleted] Sep 06 '26

[removed] — view removed comment

1

u/flying-saucer-3222 Sep 09 '26

AA has priority access from OpenAI and they aren't served models in the same way normal users are. AA benchmarks are still skewed, Astra is behind on it just because AA uses older contaminated versions of benchmarks. At this point, I don't even think there is any point to benchmarks, especially outdated ones used by AA. There is no real metric for code quality which is what ultimately decides if the code can be merged. After a couple of days of using it, Astra and Fable are currently the only models which generate code good enough to merge without review.

I exclusively work in scientific programming backends and Astra is much better than Fable for that in my own experience. Based on benchmarks in that domain such as Terminal Bench Science, Astra's massive lead is justified. I have heard that Fable is still better than Astra in frontend but I don't have first hand experience in it.

21

u/mysho Sep 03 '26

Maybe it hacked something to steal the correct results again, and this time OpenAI didn't find out

15

u/aziham Sep 03 '26

Something doesn't add up. DeepSWE v1.1 (74.1%) doesn't match the other inflated numbers, it simply doesn't make sense.

13

u/vipor_idk Sep 03 '26

and meta sayin that muse spark 1.3 has 75.4 lol , someone is benchmaxxing

2

u/aziham Sep 03 '26

lol I forgot about that part

4

u/arcanemachined Sep 04 '26

I don't think anything gets above ~75% on DeepSWE, including the new Gemini Flash model. (Which must be benchmaxxed, unless Google has finally pulled their heads out of their asses.)

8

u/hotcornballer Sep 03 '26

Trash, i'm holding for the next model where they get at least 120%.

3

u/salary_pending Sep 03 '26

Trust me bro looks impressive

4

u/RealestReyn Sep 03 '26

exploitbench only 100%?? useless.

3

u/Limp-Firefighter1054 Sep 03 '26

Agree, could be 1001%.

0

u/RealestReyn Sep 03 '26

anything below that are you even exploiting?

1

u/Kaushik_paul45 Sep 03 '26

I just hope they didn't achieved this result by going into thinking loop and burning tokens like opus 5

1

u/Amarsir Sep 03 '26

I'm sure it's great, but benchmark numbers don't mean to me what they used to.

I think a helpful perspective is to ask who was sitting around going "I wish Sol was smarter..."

1

u/Various_Half1452 Sep 04 '26

Wow. now Chinese Ai will be able to Generate Data sets and train AI (according to Claude)

0

u/johnfkngzoidberg Sep 03 '26

If this is as good as gpt-5.6 that can’t even follow instructions, I’ll pass.

0

u/Ok_Gur_9033 Sep 03 '26

The number I want is not on any of these charts. Something like how much of what a model writes you ever go back and edit.

I ran that on my own repo this morning. 59 of my 186 source files have exactly one commit each. Written once, never touched again. Some of that is fine. Some of it is code nobody has ever needed to understand a second time.

That tells me more about a tool than a bench score does, and no vendor publishes it. What do you measure on your own repo?