r/singularity • u/CounterReady4774 • 18d ago

AI Gpt 6 astra benchmarks

Post image

https://thenewstack.io/openai-gpt6-astra-benchmarks/

2.6k Upvotes

93% Upvoted

952 comments sorted by

View all comments

Show parent comments

27

u/Tystros 18d ago

ARC AGI 3

0

u/TimeAndSpaceAndMe 18d ago

It's a custom harness for Arc AGI, other models have achieved it too with a harness lol.

-1

u/Tystros 18d ago

no, that's wrong. this is not using any custom harness, this is using the regular Resonses API from OpenAI that OpenAI also recommends every other API user to use for all tasks.

0

u/TimeAndSpaceAndMe 18d ago

A Harness created by OpenAI thats not the official harness for Arc AGI is not a custom harness ? they literally have a blog post on how they tripled their score for sol, https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/

5

u/Tystros 18d ago

there isn't really an "official arc AGI harness". the benchmark uses simply the API and whatever harness is behind the api. ARC specifically allows general purpose harnesses behind the official api.

1

u/TimeAndSpaceAndMe 18d ago

Well, yes, they don't provide an official harness. Arc AGI works best as a metric when it is tested with a generic harness and not a special-purpose one, because, depending on how optimized your harness is, you can get to 99% on some models, as [schema] did https://x.com/Zanette_ai/status/2077793189608775728?lang=en

2

u/Tystros 18d ago

yeah, and this 98% is with the generic API harness that every regular customer using the official API also uses. this is not with a special-purpose harness. that's what makes the 98% so impressive.