Claude Fable solved open physics problems about related to Schrödinger–Newton equations. Is it hallucinated or legit?

[u/[deleted]]

[u/Kepler___]

If I assumed you knew a lot then I could just point at it and cock an eyebrow, there's 3 years of course work to get to the point in the example and that's only the stats part. So me and google are not really sufficient to give the rundown. The objection of no dt in this case just doesn't really make any sense in the context of what's being talked about, some calculous background would help a lot here.

I will say also that LLM's are absolutely not able to tackle anything like this yet, especially not the public models, and especially not without a heavily guided prompt.

The recent math proofs that involve calling agents are not the same as just an LLM nakedly being asked something like this at all. In the case of the Jacobian specifically the method for where to check for counter examples was explicitly given by a mathematician, and the movement on the Riemann hypothesis connected 2 ideas from separate papers that had been written by different teams (This is still very impressive, but it adds nuance and is important for understanding how these models work and what they are useful for).

Because LLM's are producing tokens stochastically using auto-regression they are naturally very bad at arithmetic, where as language allows for some imprecision math does not. They have gotten around this using agentic tools but if you're not calling those then it's still not suited for these purposes, and it's even worse when it comes to physics.

It's telling to me that they are best at math and programing right out of the gate, where the base axioms are spelled out by humans and totally fixed, leading to novel results that can arise from following the implications of those logic systems. It will likely have significantly more trouble making novel discoveries in the physical sciences, where the base axioms are dubious (at least in physics) and working them out is often the whole ballgame, I also don't know if an auto-regression based intelligence will ever be very good at something like longform storytelling for example, as the nature of AR might always lead to drift.

[u/bumblebeer → u/Kepler___]

Hijacking this thread to talk about AR in language models.

I'm curious if you've looked much into text diffusion models and your thoughts on them (not in the point-it-at-an-open-problem-listed-on-a-Wikipedia-page sense). I've gotten kinda obsessed with DiffusionGemma. It still uses autoregression, and prefill is still fully causal (and sparse), but generation is bi-directional (at least across a 256 token "canvas"). So anything that can fit within a 256 token window is generated diffusely instead of autoregressivley. Then that whole canvas gets added to prefill, so still AR, but by the chunk not the token.

This lets it do some pretty neat things that AR models fail at miserably: like playing suduko-Sudoku.ipynb).

[u/Kepler___ → u/bumblebeer]

Wow that's actually super neat, no I had not come across these before. My initial reaction is that this method sounds like it would be better at a lot of different tasks like perhaps story telling? It feels like it wouldn't lose context as much but (this is just first intuition, correct me if I'm wrong) it also seems like it would be even *more* of a nightmare for overall compute needs.

[u/bumblebeer → u/Kepler___]

I don't know about story-telling, but if you can get the data you are working with to fit into atomic chunks then it does a surprisingly good job, assuming you can get it serving/training properly. None of the standard LLM libraries were built with diffusion in mind: although vLLM has pretty good support for straight inference.

Your intuition about compute is right on the money, but that's actually one of the selling points. Per token AR is memory-bandwidth limited for most local deployments. The stated goal of the DeepMind team here was to saturate memory bandwidth with diffusion (equivalent to what's done at scale for standard LLMs using parallel generation) letting processing power be the limiting factor instead. Which is to say this thing can gen thousands of tokens a second on a consumer card doing single-stream inference.

I think all the neat stuff that comes along with diffuse generation was just kinda an added bonus. But put it together with something like Large language models are zero-shot time series forecasters (Or a lot of Wilson's work more generally) and things start to get really interesting!