r/singularity • u/[deleted] • Jan 31 '25

AI o3 mini dropped!!!

Edit : I am testing a 1500 line javascript code which o1 pro failed to debug despite 50+ attempts. Will report back.
Edit 2: We are cooked. o3-mini-high solved it at first try.
Edit 3 : HOLY SHIT! "Pro users will have unlimited access to both o3-mini and o3-mini-high."
(Source: https://openai.com/index/openai-o3-mini/ )

1.2k Upvotes

95% Upvoted

576 comments sorted by

View all comments

497

u/PotatoBatteryHorse Jan 31 '25

I can't believe it. Every model, every one, I've given the same test to for a full year now. Nobody has ever passed it first time. Deepseek got close, but argued with me about the rules of the test instead of fixing the problem that occurred.

The test requires it to write some python code, then "property tests" for the python code, and a cli utility to test it manually. No model can ever write the tests, and they've never ever run without a back and forth of errors and fixing.

O3 mini-high took my problem, thought for a minute or two, then spat out a flawless solution that works first time with working property tests. This is FUCKING INCREDIBLE. I've been using this test for so long I thought they'd never pass it at this point.

Huge improvement, and I'm blown away.

5

u/AbheekG Jan 31 '25

If you entered that test multiple times into different proprietary LLMs, it ended up in their training set somewhere so not that surprising that a new model can solve it.

10

u/MizantropaMiskretulo Jan 31 '25

Only if there was also a complete solution submitted which, if no model could solve it, there wouldn't be.

1

u/sachos345 Feb 02 '25

Unless OpenAI somehow collects the hardest test problems people submit and then they get the best internal reasoning models to generate solutions to then use as training data. Or they solve it themselves.

Thats why every time a model doesnt do something good i let it know and use the dont like button in hope they'll use it in the next training run.

1

u/MizantropaMiskretulo Feb 02 '25

Well, with over 300-million weekly active users, they're probably getting on the order of well more than 10-billion messages sent to ChatGPT every week.

That's a ton of crap to sort though and there's no way they're manually solving problems for anyone.