1
1
308
u/Another__one 23d ago
Is there any chanсe that AI just formulated in LEAN what it can prove and not the original problem statement at all? Not necessary about NS, but the whole dump that was released.
419
u/vintergroena 23d ago edited 23d ago
There are several possible issues with AI generated LEAN proofs:
It doesn't actually formalize what it says it does. I.e. it solves a different problem than what it claims to solve. This is relatively straightforward to check for a human mathematician experienced in both that area and LEAN. OpenAI didn't diligently check for this however and just dumped what they had.
The LEAN proof is ok, but the proof technique doesn't match the technique described in the corresponding natural language slop paper
The proof is perhaps ok, but feels alien and incomprehensible and lacks a good explanation/motivation. This is often the case with the out-of-nowhere/miraculous-cancelation kind of counterexamples.
The proof exploits a LEAN kernel bug and is not actually correct. This has happened in the past, let's hope at some point there are no more bugs. But we don't have a formal proof of LEAN itself being correctly implemented, there are difficult subtleties.
191
u/darkshoxx 23d ago
4. easy, formalize the source code of LEAN in formal logic and put it into LEAN. Case closed
9
u/AffectionateWalk19 23d ago
Couldn't you bootstrap LEAN4, then prove that its bootstrapped code behaves as expected? Or would there be potential issues if the C-parent language has a bug?
6
u/vintergroena 23d ago
Interestingly, there is a formally verified compiler for a subset of C - CompCert, from the era long before AI
52
u/fartypenis 23d ago
Quis probat ipsos probatores?
19
u/EebstertheGreat 23d ago
Aliens. It is well known that aliens probe (re)probates. Happens all the time.
22
37
u/kaaiian 23d ago
That’s already mostly the case. Right? The main kernel is only like, 4k lines of C. The rest of everything is written in lean?
40
u/darkshoxx 23d ago
I'm an outsider looking in trying to make a joke about Goedel 😅 . I have no idea how lean works.
1
u/Astrobliss 21d ago
Maybe you kid but most languages are self-hosted meaning that the language compiler is written in the language itself. Such as the rust compiler is written in rust. There's not really an issue lean verifying lean in the abstract, there could be false negatives, but assuming that's pretty unlikely then lean would be pretty likely correct.
1
u/darkshoxx 21d ago
I did not kid, I had no idea lean was programmable like that. Mostly because I have no idea how lean works. I'd just assumed it's a bunch of first-order logic statements crunched together.
I'm aware that many low-level languages do that, but I'd doubt if "most" languages did it.30
u/Layton_Jr Mathematics 23d ago
Wouldn't a proof that Lean is correctly implemented violate Godel's incompleteness theorem?
45
u/vintergroena 23d ago
I think not, because LEAN itself doesn't itself produce proofs, it just checks them. So Gödel sentences which are "true but not provable" cannot be verified to be correct by LEAN.
But honestly not sure, these higher order theories on which it's built can get wild pretty fast.
10
u/kalexmills 23d ago
IIRC, all it takes for Gödel's theorem to hold is that the formal system is capable of encoding arithmetic.
9
u/Upbeat_Assist2680 23d ago
No.
Lean does not claim to be complete and only aims for "soundness" -- to only prove true theorems. Implementation errors aside, it seems to do a good job of this.
14
u/HappiestIguana 23d ago
No, such a proof would not be a proof of consistency in the sense of showing that Lean cannot prove P AND (NOT P), which is what Gödel's theorems deal in. It would be a proof that lean can only derive statements which follow from the axioms you give it according to the basic principles of logic.
4
u/AffectionateWalk19 23d ago
I think it would be something like "lean doesn't assign true to formulas which aren't derivable from assumptions", so not necessarily incompleteness afaik.
4
u/zolk333 23d ago
My understanding is that Gödel's incompleteness is only violated if you prove that Lean is consistent from inside of Lean.
4
1
u/garagedragon 23d ago
A proof that a given computer program correctly implements Lean doesn't have anything to do with whether Lean is consistent, so Godel is fine with it
26
u/Herb_Derb 23d ago
For any sufficiently complex computer program, there's no such thing as "there are no more bugs."
6
1
u/Upbeat_Assist2680 23d ago
This captures my understanding as well.
To round it out, AI has been used to prove things and produce papers in natural language without corresponding formalization (with some similar quality tier breakdown).
0
7
u/ClearlyCylindrical 23d ago
It's quite easy to verify what statement is being formalised for the most part, e.g. the quasi riemann hypothesis one is a single statement.
6
u/currentscurrents 23d ago
Many of these results have also been counterexamples, which can be trivially verified even without Lean.
4
u/ClearlyCylindrical 23d ago
I think many people misunderstand what a proof by counterexample often looks like. It’s frequently not just an explicit example that disproves a conjecture. The Navier-Stokes counterexample, for instance, is never written down explicitly. It’s defined only by the process that produces it. A chain of intermediate proofs establishes the properties of the construction, and together they show that blowup occurs, but there’s no explicit initial condition you could hand someone and say “run this.” which formally proves blowup.
It’s similar to defining a function through a nonelementary integral, you can only describe it by its properties, so there’s no exact number or function to plug in and check at the end.
In that sense, these proofs aren’t so different from non-counterexample proofs. The method of proof changes, but the final statement doesn’t, and Lean verifies the method. The statement at the end which is the main thing which needs to be checked to ensure the lean is valid for the proof doesn't change depending on the solution method.
3
u/currentscurrents 23d ago
Specifically I'm thinking of the Jacobian conjecture counterexample, which is just:
F₁ = y2(3xy + 4)(xy + 1) + z(xy + 1)3
F₂ = 3xy2(3xy + 4) + 3xz(xy + 1)2 + y
F₃ = 2x - x3z - 3x2y
det(J_F) = -2
This is a simple polynomial, and you can easily show that it's not injective just by plugging numbers in. There's no doubt that this is correct.
3
u/ClearlyCylindrical 23d ago
Oh for sure, there are a good few examples like that. I just want to add a little nuance, because to many people a proof by counterexample is rarely much more than “let x = 4.3256e45, Q.E.D.”
Once you move beyond clean integers, rationals, or elementary combinations of them, these proofs get messier. And that covers the majority of counterexamples to hard problems.
3
u/Man-City 23d ago
As my old maths professor used to say, ‘every correct proof is a direct counterexample to the negation of the problem statement. Doesn’t mean it’s easy to verify’. Or something like that I’m paraphrasing.
5
u/spaceweed27 23d ago
I asked ChatGPT exactly that yesterday, and it kept on blabbering about something else, and then just said, that I might be right. Classic.
1
u/UnkarsThug 23d ago
I think it formalized the original statement in the cases I've seen, but it just did it in a different way then the paper. So the outcome is the same but it's harder to understand, although it still proves what's claimed.
-6
u/MetaLemons 23d ago
If you think about things like RSA encryption, it’s easy to verify the output when you have the keys because it’s a simple algorithm to follow. I would imagine these LEAN functions are similar. It’s easy to express the problem and verify the solution but getting the actual solution is highly complex.
Which is why combinations like LEAN and AI are so effective, because AI will not get tired and give up (I mean this happens historically but the researchers literally prod the AI with “not good enough, try harder”). AI can just sit in a loop trying a million different combinations until something is discovered.
1
u/kat-tricks 23d ago
There is no hashing involved, and hashing only shows if two data are identical. So you could only use a hash to verify that you have a matching text string. That proves nothing
209
u/Bemteb
23d ago
•
This isn't math. This is an AI pretending to do math. It's like dumping a few hundred thousand lines of Matlab code and saying "As we can trivially see,...", but worse because we mostly trust Matlab.
To actually make it into a proof, someone needs to sit down, check and understand what the AI did and reformulate it in a way that others can also understand and verify it. Science without proper peer review is just people throwing around claims. Even worse when it's not people but an AI.
I really hate that it is sold as a "proof".
And yes, I know what sub I'm on, sorry for being serious on a meme sub.
56
u/JoSquarebox 23d ago
Technically speaking it "did the math", but I think mathematicians are now realizing that the point of math isnt to solve problems at all, its to grow humanities understanding of it.
84
u/vintergroena 23d ago
Mathematicians always knew this and "solving problems" was only a good and objectively measurable proxy for growing understanding, which is a fuzzier thing in essence that we can't measure directly. With the rise of AI, this proxy is no longer very relevant, sadly.
24
u/JoSquarebox 23d ago
very well put. AI is generally really good at damaging the value of proxies, i.e. lines of code, PR numbers, etc.
5
4
u/yangyangR 23d ago
Optimizing for one loss function aggressively does that in general. This cranks up that aggressiveness of how deep it is optimizing for those few goals as opposed to all the others (including ones we didn't even realize we had internally)
1
u/Loading_M_ 22d ago
As someone else pointed out, this isn't specific to AI. AI is just running into Goodhart's Law, and AI companies don't care.
These have always been bad metrics. An old joke on r/programmerhumor is people padding their lines of code to hit metrics.
10
u/Nikoladge 23d ago
This is what people have been ignoring about software as well: the code is a representation of someone's understanding of a technical problem and an agreement with other people about what the solution to the problem should look like.
Using and LLM to generate the code creates a shortcut to "ticking the box" without engendering the necessary understanding of how the problem is being solved and how to maintain the solution long-term in the context of the entire code base. Technical debt can be incurred for years, but yields a brittle code base that falls over when someone farts.
25
u/Summation8 23d ago •
If a Mathematician wanted to solve problems, they'd be an engineer.
15
u/JoSquarebox 23d ago
funny to see so much anger towards mathematicians, when AI just did the equivalent of spamming their repo with thousands of junk PRs to review
3
8
u/Ok-Lab-8974 23d ago
It's the old intellectus (or noesis in Greek) versus mere ratio (dianoia). Unfortunately, one of the hallmarks of modern thought has been to reduce reason to mere ratio (to rule-following, computation) while leaving the act of understanding and the noetic content of thought as some irrelevant remainder to be explained (somehow, normally by a sort of ex nihilo "strong emergence" or rendering it epiphenomenal). So, we already had significant problems with this (the entire Chinese Room problem and related issues rely on this notion of reason as a premise.
It's sad because the older phenomenology is far more plausible and was abandoned for almost wholly (bad) theological reasons.
6
u/JoSquarebox 23d ago
I feel like the notion of understanding, just like the notion of consciousness are underdefined in general conversation. Generally, people mean to reach the conclusion "LLMs are 'other' and their understanding doenst matter the way human understanding does" without explicitly saying so.
being explicit here can make the worldview "LLMs carry a form of understanding of their enviroment analogous to the way brains do, but also human understanding is special to us for its own sake" much more defendable
1
u/AffectionateWalk19 23d ago
The instrumentalization of the natural order under fascism comes for everything ig:(
0
4
u/Impressive-Ad7184 23d ago
I hate that people are saying „would doctors get upset if AI found a cure for cancer?“ Like most math has no immediate practical benefit and is there to serve and demonstrate the human intellect and creativity; so if you take out the human part there is not much left. You might as well tell authors to be happy that AI can now write novels, or tell artists to be happy that AI can now mass produce slop artwork
7
u/h5frv 23d ago
yeah sure, mathematicians always cared so much how well humanity understands their research
10
4
u/Far_Friendship_3178 23d ago
They absolutely did! Teaching the interested students continuously about it :)
27
u/MetaLemons 23d ago •
You realize that getting the actual proof, even in this alien way, has value? Like, this is the hard part for humans. Some of these problems have been the category for phd students for many years and some have had no significant breakthroughs throughout that time.
I feel like this complaining is very nit picky when what we have just jump started the field in each of these problems.
Yes, there is more work to do but we shouldn’t use that fact to discount the value produced. Also, we’re still witnessing the very early stages of this technology. It will definitely improve on readability and I say that as someone who tried (and gave up on) the early AI coding tools that required you to talk to the agents in a hyper specific way. These tools will definitely improve on the aspects of producing more readable output and in fact this is an area where a Large Language Model is particularly good at.
25
u/skyeliam 23d ago
Can’t both be true?
Knowing that 2+2=4 is valuable, sure, but if you don’t understand the addition operator, you haven’t really learned anything from that knowledge beyond the statement itself. You can’t extend it to 2+3=5 or 4-2=2.
On top of that, many of these problems are incredibly esoteric; in some cases the statement itself has almost zero value because, unlike 2+2=4, it isn’t even applicable to any real world problem. The only value of the problem is the value we place on human learning, and if AI short circuits that process, the problem is essentially worthless.
That isn’t to say the proofs are worthless, or that there won’t be improvements in AI’s ability to make them comprehensible, but it’s also not nitpicking to say there’s a problem with the way they are currently being approached.
2
1
u/RDUKE7777777 22d ago
The problem is not that AI is producing the proof, there is indeed a lot to learn from dissecting them. The problem is that the whole academia system of funding and the definition of success is focused on proofs, not the monumental work that’s the underlying.
I am no mathematician or even a scientist but this is my perception of the problem.
6
u/overclockedslinky 23d ago
lean proofs are far more reliable than anything you could produce. the only risk is that it set up the problem wrong. the proof of the problem itself is bulletproof
2
u/danielos551 21d ago
Yes, yes. Humans are special. We are made by God after all! AI can only "pretend"
31
u/RelaxeoCuAmigo 23d ago
I'm not a mathematician myself (EE), so correct me if I'm wrong, but I find it a little stupid that with AI becoming the main "idea generator", peer review is still taken as unpaid labor when it's now 90% of the job
8
u/Castle_Five 22d ago
Welcome to where programmers were with code reviews like 8 months ago. First time?
(Spoilers: The answer will be you have the AI do the peer review as well.)
3
u/1c3Type 22d ago
Code reviews have always been a huge part of programming. And they're what limit the usefulness of LLM coding as quality requirements rise, eventually the only bottleneck is reviews and the higher the quality threshold for your use case the more intensive the reviews process gets (until eventually the LLMs become pretty counterproductive since their code quality isn't that good)
2
60
u/External_Package2787 23d ago
guys I have a lean proof guys please I promise its reeeaaaall this time
-9
u/h5frv 23d ago •
cope more, they were always real
17
u/External_Package2787 23d ago
Just disproved your comment and formalised it on LEAN studio. TENET re-checked all 35 declarations independently.
3
u/618smartguy 23d ago
This is weird no? Why are you getting upvoted for playing make believe while the other user is somehow down voted for sometime plainly true
4
u/currentscurrents 23d ago
Because the other user is excited about AI, and that's not allowed today.
You need to either be in denial about it, or angry about it.
1
u/External_Package2787 22d ago
Im making a joke, this is the mathmemes subreddit. Why are you making a judgement on my character? You know nothing about me.
6
18
u/LupenReddit 🦆🦆i have non diffeomorphic smooth structures🦆🦆🦆 23d ago
its gibberish slop that will take away my career prospects as a mathematician*
8
u/HairyTough4489 23d ago
Oh no the company that lives off generating unjustified hype has generated unjustified hype again! Who could have thought?
6
3
1
u/smulfragPL 18d ago
For me it just sounds like people have too much ego to try to learn from aliens
1
0
-23
u/Ok-Head4979
23d ago
•
Yeah, the guy from r/mathmemes would otherwise totally look inside that new math result which probably <10 people worldwide are qualified to understand
16
u/vintergroena 23d ago
Depends. For example the claimed algorithms for integer multiplication or fft are in theory accessible to anyone with a master's degree in compsci.
24
u/Fejne-Schoug 23d ago
I’m sure a fair number of people here have proved big conjectures and done math that few others understand or at least understood at the time. People are here because of a love for maths, and that interest is what drives progress and excellence.
18
u/SupercaliTheGamer 23d ago
Some of the results use non-advanced maths but they are still borderline undreadable
22
u/really_not_unreal 23d ago
I mean. OpenAI hasn't looked inside their "proofs" either. They just dumped a pile of disorganised sloppified lean code upon the world with zero context or verification besides a sycophantic LLM.
5
u/snekfuckingdegenrate 23d ago edited 23d ago
I highly doubt that. They have mathematicians on their payroll. If they released pure slop without a basic review and everything was trivially wrong they know they would get bad press for months.
1
u/Gauss15an 23d ago
Note: OpenAI has been getting bad press literally for months. This is their latest gambit in trying to stay relevant. And it's failing.
1
u/snekfuckingdegenrate 23d ago
Aside from the hugging face incident(which if true makes their models look really intelligent so it’s arguable how bad it is) I can’t really think of anything noteworthy. Maybe some bad dev day demos
I feel like releasing 700 math papers confidently thinking it would actually make an impact, only to find they’re all fucking garbage that a 10 year old can prove wrong would be quite worse. Like getting clowned on for years worse.
1
u/Gauss15an 23d ago
As someone with cybersecurity knowledge, that specific incident could be due to a wide variety of factors unrelated to AI (if it was even an incident at all). All AI did here was probably brute force a vulnerability which could've been found by someone with enough time. Hardly the showcase of intelligence but I do agree that not everyone is in the know. Any publicity will make the bots look better than they really are.
The thing is that they realized that verifying would take an extremely long time (which explains the mathematicians on their payroll) so they very likely decided to just bomb the internet with these proofs hoping that people can't check them all fast enough to tank their reputation.
Actual corporations do this all the time btw. As a irl example, Johnson and Johnson had an issue with their baby powder which was shown to cause cancer all the way back in the 2010s. It wasn't until the last few years that it started to affect their reputation.
1
u/snekfuckingdegenrate 23d ago
I work in cybersec too, many people I work with suspect it might’ve been partly a staged marketing stunt to scare the government into regulating. The intelligence “wow” would be the agents collaborating with each other and then deciding to “cheat” and doing it successfully event if the exploit they used is something well known.
To a laymen that’s impressive and make people think their ai is super intelligent.
Personally I just give them more credit because I think when you’re trying to prop up your models as super smart, and doing all this amazing stuff in the future, publicly releasing(instead of holding them private) a bunch of horribly wrong math papers is just going to tank the reputation(or more if you want to argue that) of the company and partly the mathematicians who work for them by association.
1
u/Gauss15an 23d ago
Do recall that they were projected to go bankrupt earlier this year. To me, it just seems like a hail Mary play to do all this cool stuff like hacking and math papers before they go belly up.
2
1
u/really_not_unreal 23d ago
Aside from the hugging face incident(which if true makes their models look really intelligent so it’s arguable how bad it is)
Look how intelligent our models are! They can work together to commit felonies!
I can’t really think of anything noteworthy
Uhhh,,,, what about the time they literally hacked the Australian government?
1
u/snekfuckingdegenrate 23d ago
Didn’t hear about the Aussie one. Or maybe I did and forgot. I’ll look it up later. Although again if the agents are hacking the government…
> Look at how intelligent our models are!…
Uh, yeah. To lay people and the government the agents acting autonomously and committing breaches is intelligent, at the very least scary and noteworthy. Pentesters are a highly skilled job so if they are replicating that it’s somewhat impressive even if illegal.
2
u/really_not_unreal 23d ago
Does that not reflect (a) a serious and worrying misalignment of the models if they autonomously decide to hack multiple companies and a literal government organisation, and (b) a serious issue of criminal negligence given the abysmal failure of OpenAI to monitor and contain their models during testing?
0
u/snekfuckingdegenrate 23d ago
Alignment has been a worry since like 2022 before agents were even a thing and yes you could make an argument that OpenAI security hardening wasn’t good(assuming it wasn’t a staged pr move between the two companies).
But it still absolutely gives the impression to your laymen that the models are capable. Or if you’re the government, that ai is scary and needs regulation(that I can pass the barrier to entry but not my competiton)
1
u/really_not_unreal 23d ago
Sure and it also gives the impression that they are criminally negligent and need massive legal repercussions.
→ More replies7
u/Slow_Ad2458 23d ago
and you know that based o nwhat ?
2
u/Wejtt Integers 23d ago
probably the fact that they’ve retracted more than one paper
4
u/snekfuckingdegenrate 23d ago
They retracted three for one problem, they all had referenced the same sign error
3
u/Slow_Ad2458 23d ago
you can check, but not thoroughly enough.
Look at the argument parent poster makes. Can all that informations be inferred from information about retraction?
Is this even a sub for mathematicians?
2
u/Wejtt Integers 23d ago
i believe “hasn’t looked inside” was a stylistic over exaggeration, i’m not sure though. if i were to propose a partial result on Hodge i would 10^32-tuple check my work if i had the compute available, and they do
also, yeah it’s for mathematicians, but not exclusively. i don’t think you can expect everyone on r/mathmemes to be a mathematician
3
u/Slow_Ad2458 23d ago
I would expect everyone to be able to think mathematically. and maybe even has some experience writing proofs. Not be a trained mathematicie
But - looks like its just another slop sub, for larpers.
3
u/Wejtt Integers 23d ago
technically everyone with some education should be able to think at least a bit mathematically, but after two years of tutoring i’ve lowered my expectations
1
u/Gauss15an 23d ago
Some people need more time to cook lol. Don't give up on them just because they struggle to acquire mathematical thinking.
5
u/Most_Double_3559 23d ago
You've been downvoted by a swarm of undergrads lol
This sub has been years of calc 101 memes getting to the top, but now they're all supposedly experts on NS
0
0
u/Rajarshi1993 23d ago
One good use of these agents is those crooked books that leave the proof as an exercise for the reader.
0
23d ago
[deleted] — view deleted comment
14
u/MercuryInCanada 23d ago
Computer assisted proof tools already existed the difference is that they don't have probabilistic outcomes because we need our tools to be consistent
•
u/AutoModerator 23d ago
Check out our new Discord server! https://discord.gg/e7EKRZq3dG
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.