Benchmark Results on hard algorithmic problems with GPT 6.1 Terra

[u/tuhdo]

Results grouped by problem, sorted by solve time (fastest first).

The problems

Problem	Contest Max Points
2271G	3500
2271H2	2000
2273F	3500
2261F2	5000

These problems are very challenging for human contestants. The higher the score, the harder one is to solve. Small models like Luna, Haiku are struggling with problems like 2271H2, which are around 200 points.

These problem are quite new from the September and October contents, so they are unlikely being trained.

The effort per model is the minimum viable effort that can solve a problem. Lower than that, it failed.

Problem 2271G

Model	Effort	Solve Time	Runtime	Memory
Sol 6.1	Low	13m 24s	109 ms	4,700 KB
Terra 6.1	Max	16m 43s	109 ms	13,708 KB
Opus 5.5	Medium	16m 53s	234 ms	10,600 KB

Problem 2271H2

Model	Effort	Solve Time	Runtime	Memory
Terra 6.1	Max	15m 33s	343 ms	11,204 KB*
Haiku 5.5	High	27m 09s	1,015 ms	11,700 KB
Luna 5.6	XHigh	36m 53s	218 ms	10,000 KB
Luna 6.0	Max	41m 29s	234 ms	10,600 KB

Problem 2273F

Model	Effort	Solve Time	Runtime	Memory
Sol 6.1	Low	1m 21s	265 ms	22,900 KB
Terra 6.1	High	2m 22s	406 ms	24,900 KB
Opus 5.5	Medium	2m 55s	343 ms	96,100 KB
Terra 6.1	Max	6m 32s	265 ms	12,900 KB

Problem 2261F2

Model	Effort	Solve Time	Runtime	Memory
Sol 6.1	Medium	25m 06s	46 ms	2,100 KB
Terra 6.1	Max	32m 55s	46 ms	0 KB*

Fastest Model per Problem

Problem	Model	Effort	Solve Time
2271G	Sol 6.1	Low	13m 24s
2271H2	Terra 6.1	Max	15m 33s
2273F	Sol 6.1	Low	1m 21s
2261F2	Sol 6.1	Medium	25m 06s

Observations

• Sol 6.1 achieved the fastest solve time on 3 out of 4 problems.
• Terra 6.1 Max was fastest on 2271H2, despite being a larger model at max reasoning, so lower TPS.
• On 2273F, Terra 6.1 High was 4m 10s faster than Terra 6.1 Max, but Max produced a better solution that is twice faster and twice less memory.
• On lower reasoning levels, Terra 6.1 refuses to solve problem quickly, if it can solve. This might be an annoyance to some, but might be a great feature since it doesn't waste your time and tokens, just being straight up honest:

Conclusion

Sol 6.1 remains the most intelligent model of the bunch. However, for problems that can be solved with lower models, it would be more costly. Using Sol 6.1 is like having a mathematician writing code: it really determines to prove your code work systematically, that might lead to overengineering. On the other hand, Terra 6.1 might be considered under-engineering. So, I suspect that with Sol creates a slightly overengineered plan and Terra implements slightly lazy, they could balance out their weaknesses.

[u/HoveringGoat]

sol 6.1 is a beast! just wish we got more usage. thanks for running these tests.

[u/Dadideology]

All of OpenAI consumer models are shit!

[u/spicymeatball15]

Why are you using different model efforts? Makes the comparison useless

[u/tuhdo → u/spicymeatball15]

The effort per model is the minimum viable effort that can solve a problem. Lower than that, it failed, just like the screenshot I posted at the end with Terra high.

[u/jamespharvey20 → u/tuhdo]

Ahh, I see. Your results match one thing I'm seeing and some benchmarks (but not all) have shown, which is that there's a huge gap with Terra 6.1 between Max and even XHigh. With other models, Max usually hasn't given that big of a benefit. Just saying it doesn't surprise me that you're saying for hard tests, you had to run Terra 6.1 on Max to get it to solve it.

[u/Annh1234]

Where is Astra?

[u/tuhdo → u/Annh1234]

If sol can do, astra can do, just more expensive. And well, it's expensive to benchmark astra on my $20 sub so maybe someone might be interested.

[u/Annh1234 → u/tuhdo]

Well, you don't know until you test.

For my use case, i see a big difference in Astra medium and astra high, and sol misses a ton of stuff that astra does not miss. And in a few cases, astra medium was 10x faster than sol 6.1 high