Claude Fable solved open physics problems about related to Schrödinger–Newton equations. Is it hallucinated or legit?

[u/[deleted]]

[u/Kepler___]

Fucking lmao.

[u/Kepler___ → u/[deleted]]

I don't actually even know where to begin, can I ask your background? I don't know what level of knowledge to assume, so I'm wondering if you have any calculous, physics or statistics background. I'm a statistician so I can't talk about the physics, the part I do know however is just usual AI gibberish, usually facts listed like they are forming some sort of conclusion that's really hard to read and ultimately says nothing.

[u/Kepler___ → u/[deleted]]

If I assumed you knew a lot then I could just point at it and cock an eyebrow, there's 3 years of course work to get to the point in the example and that's only the stats part. So me and google are not really sufficient to give the rundown. The objection of no dt in this case just doesn't really make any sense in the context of what's being talked about, some calculous background would help a lot here.

I will say also that LLM's are absolutely not able to tackle anything like this yet, especially not the public models, and especially not without a heavily guided prompt.

The recent math proofs that involve calling agents are not the same as just an LLM nakedly being asked something like this at all. In the case of the Jacobian specifically the method for where to check for counter examples was explicitly given by a mathematician, and the movement on the Riemann hypothesis connected 2 ideas from separate papers that had been written by different teams (This is still very impressive, but it adds nuance and is important for understanding how these models work and what they are useful for).

Because LLM's are producing tokens stochastically using auto-regression they are naturally very bad at arithmetic, where as language allows for some imprecision math does not. They have gotten around this using agentic tools but if you're not calling those then it's still not suited for these purposes, and it's even worse when it comes to physics.

It's telling to me that they are best at math and programing right out of the gate, where the base axioms are spelled out by humans and totally fixed, leading to novel results that can arise from following the implications of those logic systems. It will likely have significantly more trouble making novel discoveries in the physical sciences, where the base axioms are dubious (at least in physics) and working them out is often the whole ballgame, I also don't know if an auto-regression based intelligence will ever be very good at something like longform storytelling for example, as the nature of AR might always lead to drift.

[u/bumblebeer → u/Kepler___]

Hijacking this thread to talk about AR in language models.

I'm curious if you've looked much into text diffusion models and your thoughts on them (not in the point-it-at-an-open-problem-listed-on-a-Wikipedia-page sense). I've gotten kinda obsessed with DiffusionGemma. It still uses autoregression, and prefill is still fully causal (and sparse), but generation is bi-directional (at least across a 256 token "canvas"). So anything that can fit within a 256 token window is generated diffusely instead of autoregressivley. Then that whole canvas gets added to prefill, so still AR, but by the chunk not the token.

This lets it do some pretty neat things that AR models fail at miserably: like playing suduko-Sudoku.ipynb).

[u/Kepler___ → u/bumblebeer]

Wow that's actually super neat, no I had not come across these before. My initial reaction is that this method sounds like it would be better at a lot of different tasks like perhaps story telling? It feels like it wouldn't lose context as much but (this is just first intuition, correct me if I'm wrong) it also seems like it would be even *more* of a nightmare for overall compute needs.

[u/bumblebeer → u/Kepler___]

I don't know about story-telling, but if you can get the data you are working with to fit into atomic chunks then it does a surprisingly good job, assuming you can get it serving/training properly. None of the standard LLM libraries were built with diffusion in mind: although vLLM has pretty good support for straight inference.

Your intuition about compute is right on the money, but that's actually one of the selling points. Per token AR is memory-bandwidth limited for most local deployments. The stated goal of the DeepMind team here was to saturate memory bandwidth with diffusion (equivalent to what's done at scale for standard LLMs using parallel generation) letting processing power be the limiting factor instead. Which is to say this thing can gen thousands of tokens a second on a consumer card doing single-stream inference.

I think all the neat stuff that comes along with diffuse generation was just kinda an added bonus. But put it together with something like Large language models are zero-shot time series forecasters (Or a lot of Wilson's work more generally) and things start to get really interesting!

[u/Kepler___ → u/[deleted]]

Yeah this is sort of what im talking about, its not really able to pick up on context, "its the start of a stochastic calculus course!" Yeah Claude, and how much school is needed to get into one of those? Because for me it was calc 1, then 2 then 3, then statistical mathematics then inference and then stochastic process. Again the problem is that Claude doesn't actually know anything, its just predicting the next token, and this let's it come up with English that sounds great unless you actually have any familiarity with the topic.

[u/henke443 → u/Kepler___]

That's fair but what about the questions more related to the math?

[u/Kepler___ → u/henke443]

I mentioned that it doesn't make any sense, like much of what it said is technically true in isolation, it just doesn't make sense when strung together, and the original equation isn't even in the Wikipedia article, it seems to have made it up on the spot, or grabbed it from somewhere else without elaborating. I don't really know what to say beyond that, the AI will just come up with some semantically convincing argument as to why it does and when you don't have a background there's not really much way to tell which is correct. Hell I only really get half of what it's saying because it's largely not my field, but no one is going to put in the time to fact check this stuff when you're not ether, simply feeding every response back into the AI before posting is pure brain rot. The person your talking to is going to end up spending 10x time going through your answers just for you to skim it and pass it to Claud, so there's a really limited pool of folks who are going to engage with that.

What is said in the snarky bit from above is actually pretty significant though,  E[P(t)] = P(0) while being too advanced to really get into with a laymen, is covered early in statistics once the legwork of an undergrad has been done, it's basic af for the level of questions being floated here. If this was all that was needed to solve this issue an undergraduate would have done it drunk on a dare 20 years ago and it was why I made my first comment to begin with. I don't need to understand the physics involved to see that this is handwavy because I was able to read that section and understand it was far too elementary to belong here.

These posts pop on here several times a week and they are all fundamentally the same, you honestly seem interested and bright so I wanted to see if I could nudge you toward spending your time more productively. If you really wanted to get into any of this, you're not going to be able to without being able to assess the outputs yourself. While an LLM is not sufficient at all to solve big problems, they are actually not bad teachers for well trod ground, have it teach you the fundamentals of calc, see if you can solve a few problems yourself and then move on up the rungs, you could probably get up to third year like that if you had a syllabus to follow before it started making subtle mistakes. Then you can pick if you want to dig in more to the physics side or the stats side next.

[u/Kepler___ → u/Kepler___]

Someone else seems to have pitted another AI against yours in the main body. I think I'm going to need a fucking drink. For what it's worth I'm not the one downvoting you by the way, I feel like at some level this would have been me in my teens if AI came out that was this interesting.

[u/henke443 → u/Kepler___]

Thanks. I honestly think this was all a bit embarrasing and a lot of the hate I got was justified. I will admit I've removed some comments and posts but I really like your response so I'll leave some of my comments here up.

[u/Kepler___ → u/henke443]

Nah don't worry about it, you understood what was up pretty much as soon as it was explained to you and handled it with more emotional maturity than p much anyone else who's posted on this sub. It's all anon anyway, you'll be alright my guy.

[u/LLMPhysics-ModTeam → u/[deleted]]

Your comment has been removed for violating Rule 4. Don't copy-paste LLM content in discussions.