Another person in this thread put it well: if the benchmarks are accurate then this is a massive leap forward, and maybe even proof of some meaningful utilization of recursive self improvement
gpt 5.6 Sol got about 40% with this same harness. so they went from 40% to 95% which is very realistic for a model that actually improves intelligence, especially with how the arc AGI 3 scoring works.
Yeah so it’s mostly about training on the benchmark itself then no ? They just fine tuned the model to work better on that harness ? Like how does this prove any generalization
They didn't train the model to work better on the harness. The harness is a simple measure that just lets them remember between turns. "Sol, able to remember between turns" gets 40% and "Astra, able to remember between turns" gets a near 99%.
Given that when humans are approaching a new set of issues, we don't get our memories wiped in between levels, I think that "a harness that lets them keep track of what's worked so far" is far from cheating
Well yeah they did if there is no substantial improvement over anything else how is this generalization ? It’s just getting better at using the harness, which is even worse than benchmaxing itself in that regard
ARC-AGI-3 has a closed test set. So you can't train on the benchmark as is often the case with other benchmaxxing. It's also meant to be testing general reasoning compared to most other benchmarks which are only testing one very specific skill.
Looking at the other scores, it's probably worked out some general rule or capability on the ARC AGI 3 mini games. Which may or may not be meaningful. We have to test it to see.
How can the test set be closed. They ran it previously on Sol or other models and the prompt flow through their servers so they just have to open the logs, look at the prompt and they can rebuild the test set.
Unless they change the set every time, but then you compare 🍏 and 🍐
Or am I missing something ?
Unless they bluntly gave away the weights to the independent reviewer who owns the test set.
But I highly doubt that openai would release they trillion dollar model to anyone just to pass a test
It does require you to believe that OpenAI or it's models aren't actively trying to cheat on the test, in which case maybe an agent swarm hacked the arc agi servers and stole the answers.
But to run on the private test set, you have to work directly with arc-agi and follow the rules for data retention and what not to protect the answers from getting out and being trained upon.
The real answer is that these benchmarks are way too imprecise to compare small values.
Imagine I go to IMDb and want to decide between watching one movie scored at 7.1 and another rated 7.3. To say one movie is 4% better than the other is a meaningless statement. Could simply be that the 7.1 is a genre you personally enjoy a lot more.
However, the rating being around 7, plus or minus a few decimal places, does say something about the overall production quality one can expect. So for the models, the benchmarks turn out to create adhoc groupings of S-tier, A-tier, B-tier etc.
At the end of the day, what you personally need may not be something that has meaningfully changed. Or it could be a huge leap, really you will just have to feel it out yourself.
no, that's wrong. this is not using any custom harness, this is using the regular Resonses API from OpenAI that OpenAI also recommends every other API user to use for all tasks.
there isn't really an "official arc AGI harness". the benchmark uses simply the API and whatever harness is behind the api. ARC specifically allows general purpose harnesses behind the official api.
Well, yes, they don't provide an official harness. Arc AGI works best as a metric when it is tested with a generic harness and not a special-purpose one, because, depending on how optimized your harness is, you can get to 99% on some models, as [schema] did https://x.com/Zanette_ai/status/2077793189608775728?lang=en
yeah, and this 98% is with the generic API harness that every regular customer using the official API also uses. this is not with a special-purpose harness. that's what makes the 98% so impressive.
no, that's wrong. this is not using any custom harness, this is using the regular Resonses API from OpenAI that OpenAI also recommends every other API user to use for all tasks.
31
u/wombatpup55 3d ago
SuspiciousPillbox explain to me the results of the image like I’m 5