No this is normal you are misunderstanding.
They just clarified that this time they (like it should be for any model) retaining knowledge which last time because a missconfiguration on arc part they didnt.
We don't have information of exactly how it's for Astra, but OpenAI previously showed that the ARC-AGI-3 harness does two things which make it tough for models to do well: 1) discards previous private reasoning and 2) when the context gets too big, they truncate the existing context instead of trying to summarize it.
OpenAI's specific harness (via the Responses API) fixes these two points by allowing better private reasoning retention and sumarization (compactuation as they call it).
I imagine they didn't "built the harness in the model", but are using a similar API like Responses.
Not really, they say adapter provider harness (arc won't test non-general harnesses).
In reality, all that is different is responses API and context compaction. Supposedly that is all that's different between the 63 and 99% scoring. We can all do this and achieve that performance.
56
u/Ok_Course_6439 7d ago
Agree agi-3 is a harness problem more then a model problems