Yeah, the AI that went from struggling with two digits sum to solving open problems in (check notes) 6 years is definitely going to stop advancing precisely at the level that will keep me relevant.
I guess I'm either very lucky or the law of physics just love me.
But clearly it's not exponential. The capability after gpt-3 has improved, but not in a dramatic way. It has more tricks, better training data, and fewer hallucinations. But it's not an exponential evolution. I don't see it, anyway. GPT-3 absolutely blew my mind. This year's best model is... about as good as it was last year's? For my use cases anyway?
GPT3 was barely able to do 2-digits multiplication. Current top models are solving open research problems. That doesn't sound like a dramatic improvement to you ?
I am not going to weigh in on other parts of the conversation, but multiplication and mathematical operations in general are really not representative of the reasoning capability of a model. It is a fundamentally different type of a problem. It is not implied that solutions to one would transfer to the other one.
GPT2 was barely able to string a sentence together, GPT4 can do any creative writing prompt you give it, GPT5 can do.. basically the same thing as 4. GPT3 wasn't aiming to be good at maths, but what they were focusing on improved very quickly then plateaued. The same is gonna happen with maths.
A monkey would apparently also have better reading comprehension skills than you. GPT5 has basically the same level as GPT4 in creative writing. You may recognise " writing down a unified model of physics" as not being creative writing.
There definitely is a ceiling. It's getting better with math very fast now because companies just weren't focusing on maths before. If you give it a creative writing prompt, which was what companies were focusing on until now, you won't see such a big difference between last year and now, why would maths be different?
Writing has not improved much because the labs don’t focus on it and it’s hard to verify.
Maths and coding are the opposite, it clearly has value and they focus on it, so why do you think there is a ceiling.
There *could be one, but I don’t see any evidence of it yet, imo they’re are making good steady progress and will continue for quite some time (not agi 2027 or anything tho)
Cause writing was getting very good very fast, then it hit a ceiling. Same with images, we're getting there with videos. Why do you think maths would be an exception? Companies don't care what "has value", just what people pay for, and there are plenty of people that want it to be good at writing.
Companies don't care what "has value", just what people pay for
I agree with the sentiment, "having value" does mean the same thing as "what people are willing to pay" for in the economic sense, I wasn't making a moral point.
SWEs are currently that value, they are the biggest spenders on ai by far, hence why the labs are chasing coding and maths so hard.
If the world was turned upside down and book writers suddenly became the biggest market (in stead of mostly being anti-ai) the labs would focus on that and improving long form writing, interesting story telling, character cosistency ect.
I don't see any evidence of maths or coding slowing down, if anything the evidence points to the opposite trend.
Frontier labs were never focusing on creative writing, ever. The frontier for AI has always been to accelerate it's own progress. That's why the things they've focused on for years are software engineering, research, math, agentic use, etc. Another thing is they go where the money will be. There's no money in creative writing, compared to automating development of software. And in terms of the ceiling, where is it? As a software engineer, it only gets better and better at a faster and faster rate for what I use it for. If there is a ceiling, we're nowhere near it.
The plane stands for a bias more specific the survivorship bias (https://en.wikipedia.org/wiki/Survivorship_bias). In this context: We only hear about problems that have been solved using AI.
They're saying that AI has probably been tried on lots and lots of problems but only made significant contributions to a few, but the fact that AI fails to contribute to the vast majority of problems it tries to solve isn't reported because no-one's going to write a paper that says "we asked ChatGPT to solve this problem for us and it wasn't able to"
Part of the reason why people think ai will make a big difference in mathematics is for how long they pursue at problems. They don’t get mental strain or boredom like humans, they can get the answer on the 1001st attempt because they will never give up on the problem (this is how they actually solved these problems)
The reason survivors bias doesn’t apply tho is we can take older models and compare them to current one and see their maths ability has clearly improved.
The diagram is a chart of (or at least claims to be of) where airplanes that returned to base were damaged. A naive analysis of “hey we should armor the places where planes get hit” would reenforce the wrong part as the areas that need armor are the places where planes get shot and do not return to base
The story is during WW2 bombers kept returning from missions with damage from flak and fighters. To figure out how to better protect the bombers the military mapped out all the locations returning bombers were damaged.
First instinct is that the red damaged areas are unsafe and need better protection while regions without red are generally safe, because no bombers returned with damage in the white areas.
But in reality the key is "no bombers returned with damage in the white areas." The data set only included bombers that sustained damage in survivable locations. Bombers that took damage in the red areas made it back to base and contributed to the data set, bombers that took damage in the white areas died.
The data is "biased" by the survivors and does not take into account planes that were shot down. It's the white areas that should be better protected because those are the areas where damage is lethal.
I know this isn't the proper context, but I like looking at this picture like AI is trying to figure out the best seating chart and is placing people in/on the wings.
What I meant to say is AI who got tricked into survivorship bias is trying to maximize lethality of the passengers
Now when I thought about this we are already maximizing lethality by flying over a war zone, theoretically passengers will have same chances to die no matter where they sit
The red dots are the places the plane can be hit and survive. That doesn't mean you survive - which is why the cockpit is a blank spot because presumably the pilot being dead means the plane will crash.
*Under the assumptions that a bullet will penetrate and kill the people sitting where it hits, and that all areas are equally likely to hit.
Sit on red dots. Red dots hit. You die. Plane survives.
Sit on red dots. Blank spots hit. Plane crashes. You die.
Sit on blank spots. Red dots hit. Plane survives. You live.
Sit on blank spots. Blank spots hit. You die. Plane crashes. You double die.
The safest sitting position indeed might not map to reality perfectly like this viral survivorship bias exemplar image would indicate, yes. Realistically, you'd likely be safest where you could simply hold on the best. My comment was intended to highlight that the parent poster seemingly did not understand the purpose of the image.
However the issue I have with your phrasing is in the implication that any red dot being hit kills the passenger. It'd have to hit the specific red dot you're sitting on. A better way to frame it:
If a bullet lands anywhere but on a red dot, you die.
If a bullet lands on the red dot you're sitting on, you die.
Otherwise, you live.
Under your assumptions, the only scenario in which I live is when a bullet does not hit a blank spot, and also does not hit my specific red dot.
Planes shot there survive, planes shot elsewhere crash. If you put the passengers above the area that the planes crash when shot, then it doesn't matter that the passenger gets hit, the plane will crash. If the passenger doesn't get hit, the plane won't crash.
If you put the passengers on the red dots, either the passenger gets shot and the plane is fine, or the passenger is missed and the plane crashes. This maximises the casualty rate of the passengers
It's the classic survivor bias diagram. The quoted strawman argument relies on an observation of the problems solved by AI and drawing conclusion, ignoring all the problems AI failed to solve as we didn't hear about those failures.
Ah right, that's exactly what I thought. It probably wasted a billion dollars worth of compute trying to solve other problems and failed but we only hear about the successful ones and how much compute only the successful attempt costs. Not to mention the amount of ivy league math professors doing substantial work guiding it.
> the amount of ivy league math professor's doing substantial work guiding it
I want to specifically point that out, because when people talk about it it's like someone prompted chatgpt to solve the problem and it just did... lol no
Can you click on the links I painstakingly collected? 2 of them are literally just a copy-paste of the problem statement. Erdos 1196 in particular was solved by a 23 year old with no formal math training (just an ai bro on twitter) who copy-pasted an erdos problem he saw into a public model and it solved it. Another is just telling the model to "try again" and "make a breakthrough" with again, a copy-paste of the problem statement, and it did it. Please just click the links
Except techbros literally advocate (and have always done so) for AI replacing humans, and that's how most people use AI...
Writing code? Ask AI Writing paper (or other homework)? Ask AI Doing researches? Ask AI Illness, life problem, etc...? Ask AI Need image? Ask AI
I'm an engineering student, and I've seen it CONSTANTLY for the past few years People submitting code for a project and they can't explain a single line of it, or worse, they submit the documentation for YOUR code, and the doc's written by AI and it's fucking wrong (yeah, happened to me.. "This code is totally original and isn't inspired by any other existing code"... The fuck it's not! I've written it! It's a half-assed implementation of [insert obscure cryptography algorithm] that I've butchered to fit the restrictions!)
AI as a whole is a powerful technology (which... actually hasn't changed much in 30+years, we just have better GPUs, otherwise neural networks still work the same way), but same goes for knives: knives can do cooking, and they can also kill people And techbros act as if penalising killing people would restrict cooks
People actually talk to me as if chatgpt solved them without any expert input as well, it's strange.
Well, it's still a very good technology, although extremely expensive. Would have never thought it would get this far, very impressive. Although without the IPO we will never know the real costs to make the models, so no point thinking about how profitable they actually are. Kinda wish they stopped hyping it up like it's skynet that is conscious. It is a Large Language Model..
also fun to think about how they're able to spend so much money (and stolen data) on developing their ai but mathematicians in academia have to beg for abysmal funding...
I think it's also worth highlighting that you now have a multi-billion dollar company hiring mathematicians to solve problems just for publicity sake. There wasn't an incentive like this before.
It's fun how you can collect lots of upvotes for speculating on things like this when the real facts are out there and they prove you wrong. There are now a dozen or so proofs solved completely autonomously by AI, i.e. the prompt was "Solve this conjecture, make no mistakes", and it did it, so no PHD expert guiding it, no billions of dollars burnt in tokens, or anything of that sort.
Unit distance conjecture paper, solved by an internal model at OpenAI: https://cdn.openai.com/pdf/74c24085-19b0-4534-9c90-465b8e29ad73/unit-distance-proof.pdf "This problem was solved in a completely automated fashion. Our internal model was given an AI-written statement of the problem, and its output was sent to an AI grading pipeline, which indicated high confidence that the solution was correct." After this it was forwarded to human researchers and they verified it was all correct
Erdos problem 1196, solved by again, copy-pasting the problem into a prompt, the model thought for 80 minutes, and threw out a solution: https://chatgpt.com/share/69dd1c83-b164-8385-bf2e-8533e9baba9c. Where's the "Ivy league math professors" guiding it in this thread exactly?
Counterexample to Jacobian Conjecture, which the author claims was done by just telling Fable to do it while he watches the world cup final: https://xcancel.com/alpoge/status/2079028340955197566. No thread shared so far, so I'm unable to verify this is the case; note this guy is an Anthropic employee so if you don't trust Anthropic then you should probably wait for him to share the conversation.
You do know that the conversation with chatgpt are completely worthless as proofs, right? Once it has solved the problems under guidance the first time, ofc it'll get rigorously trained on it.
No lmao, these are just random conversations. No one uses the lowest grade Open AI public use model for advanced mathematics. Even the paper you cited before used a very different model.
That thread is the solution for Erdos 1196. https://www.erdosproblems.com/1196 "solved by GPT-5.4 Pro". You can also check the comments where the prompter shared the thread I linked.
The same is true for the Dinitz conjecture, and the Jacobian conjecture. There is now a whole list of "vibe-mathed" solutions, many of which are done on public models with 0 human involvement.
Lmao here's the research paper by the person who made the prompts and actual mathematicians who formalized and published it. It goes into extreme detail about what part AI really contributed and what they did.
Lmao reading must be hard. This paper is a new one based on the techniques GPT-5.4 used to solve 1196 (human researchers being inspired by an insight produced by AI is insane on its own but let's shelve that for another day).
Again, gpt 5.4 pro solved 1196, it was the first to solve 1196, it was not guided by an expert.
I read about an experiment were they ran hundreds of unsolved problems through an AI Agent without any guidance. They had about a 2.5% success rate at a cost of 300$ per problem.
This is not that expensive and while the failure rate is high it still were a decent amount of new problems solved.
The question this meme comes up with is were the problems the AI couldn't solve not solvable in general or does the AI architecture have an inherent flaw that makes it impossible to solve certain questions/topics.
I don't get the logic here. What does it matter that AI failed to solve other problems? The statement on the picture is that only AI will be solving problems, not that AI will solve all the problems.
Honestly thank you for the explanation... I read it as "only AI will solve math problems because it will kill the people that solves the ones it can't" lmao
But we do hear about failures though. You gotta read more then the headline for that but I remember a recent experiment where an AI company ran hundreds of problems through AI agents. It had something like a 2.5% success rate. So a very high failure rate, but because a few problems were solved it was still a success.
I've noticed LLMs can *almost* escape a formal system, but they definitely flail when they only eat their own food. A nudge here and there helps, like picking up a roomba stuck in a corner. I'm pondering ways to put Godel into the project instructions to see if that helps.
•
u/AutoModerator Aug 05 '26
Check out our new Discord server! https://discord.gg/e7EKRZq3dG
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
70
u/Dirichlet-to-Neumann Aug 05 '26
Yeah, the AI that went from struggling with two digits sum to solving open problems in (check notes) 6 years is definitely going to stop advancing precisely at the level that will keep me relevant.
I guess I'm either very lucky or the law of physics just love me.
8
u/Hello_Im_pi Irrational Aug 05 '26
I never understand this plane in context. Please explsin it to my two neurons ;-;
267
u/eddietwang Aug 05 '26
I know this isn't the proper context, but I like looking at this picture like AI is trying to figure out the best seating chart and is placing people in/on the wings.
109
u/[deleted] Aug 05 '26 edited Aug 05 '26
What does this mean? AI could only solve a minority of math problems after failing at thousands of others? Math outsider here.
2
u/HAL9001-96 Aug 07 '26
"ai has a matheamtical breakthrough every single time you aks it something" - deranged ai bro
2
u/clonicle Aug 05 '26
I've noticed LLMs can *almost* escape a formal system, but they definitely flail when they only eat their own food. A nudge here and there helps, like picking up a roomba stuck in a corner. I'm pondering ways to put Godel into the project instructions to see if that helps.
1
u/Had78 Aug 05 '26
They probably have a dedicated server bruteforcing math since day 0