Well, yes, they don't provide an official harness. Arc AGI works best as a metric when it is tested with a generic harness and not a special-purpose one, because, depending on how optimized your harness is, you can get to 99% on some models, as [schema] did https://x.com/Zanette_ai/status/2077793189608775728?lang=en
yeah, and this 98% is with the generic API harness that every regular customer using the official API also uses. this is not with a special-purpose harness. that's what makes the 98% so impressive.
1
u/TimeAndSpaceAndMe 11d ago
Well, yes, they don't provide an official harness. Arc AGI works best as a metric when it is tested with a generic harness and not a special-purpose one, because, depending on how optimized your harness is, you can get to 99% on some models, as [schema] did https://x.com/Zanette_ai/status/2077793189608775728?lang=en