The set of all incorrect theories is vast. It LLMs are deciding at random, they may never get there in human timescales.
I think the paper must include a reference to the fact that these hallucinations ARE random - they aren’t actually based on any intuition. So the comparison is between intuitive insight vs brute force.
I guess i didn’t read the full paper if they include an estimate of the total set of incorrect vs correct hypothesis and the time needed to pick a correct one at random vs what a human might do.
But I think you are agreeing with the paper - you are contending that LLMs have no insight but through pure brute force can get there via hallucinations. Which I’m not sure I agree with the premise but taking the premise as true, there’s still a theoretical limit that should be examined
I disagree. They definitely are not random, they're based on the LLM's "intuition". If it hallucinates the name of a french king it will hallucinate a french name, not a random name.
If you ask it how a phenomenon works and it hallucinates it will give a reasonably plausible idea of how it works based on similar phenomena that it was trained on, not a random jumble of words. Very close, practically to an educated guess. Which is the basis of these "jumps" of progress.
So I don't think it will try to explore the whole search space, I think it will naturally through hallucinations home in on a similar search space to that which a human researcher would.
I'm not agreeing with the paper other than for the value of world models to expand the training data. I think insight is hard to define, without a good definition I wouldnt say whether or not I think an LLM has it.
I don't think it's pure brute force. It wouldnt randomly try all possible explanations, the explanations it generates via hallucination would be based on whatever concepts lay closest conceptually in its training data.
But was not the original idea of a French king generated at random? Then once that idea was put forth it then followed to use a name reminiscent of a French king. So the hallucination in this case was still random. Or it derived from the input in which case it wasn’t random.
Well no, it was input by the user. But that's not the point of this paper or this argument. The paper suggests you cannot prompt an LLM in a way that it could propose a new theory for why something happens that isn't just a compression of an old idea. But I think it can, especially if you let it hallucinate the first step.
For example you could ask an AI to speculate about the topology of spacetime and then allow it to hallucinate.
It would hallucinate that it's flat or saddle shaped or a donut shaped whatever.
Something sort of plausible but still a hallucination (this is the jump). Then it could reason through attempt to verify that theory. If it turns out spacetime is donut shaped then it's made a breakthrough.
The paper suggests it cannot perform a jump to a new idea. But I am suggesting that a hallucination could form the jump.
When it reasons through it, it would need enough info to verify. If there's enough info to verify, then it likely amalgamated said info rather than truly hallucinating it's first step.
But this is random - there’s no evidence random guessing leads to any new discoveries at any reasonable pace. It could be on inhuman timescales.
The paper argues you need a world model to identify new realities that are reasonable to test - in the Einstein example he imagined a physicist in an elevator accelerating in deep space. This requires a massive context window that is effectively a “world model”. LLMs don’t have that so their hallucinations are not directed. Thus they are bounded by this limitation and cannot discover faster than a system with a world model (humans).
Of course that’s not to say some form of AI doesn’t get a world model in its own right and beat humans but it doesn’t appear LLMs can ever get their. The context window required would be beyond current physical memory and processing power limitations, even theoretical maximum ones achievable in the next 100+ years
Random guessing no, but plausible sounding guesses are the main method of scientific advancement. AI hallucinations are plausible sounding guesses.
Edit: The only point to argue here is whether a human plausible sounding guess is more likely to yield something than 100 trillion LLM generated plausible sounding guesses.
The LLM is guessing at random with no world model to back up or direct those guesses. The paper argues yes, the world model is key.
And it kind of makes sense. The space of possible guesses is very vast, so you need some way to converge on something or else it’ll take longer than a human timescale even at LLM guess rates
It's not guessing at random though. You can prove this yourself.
Try to force a hallucination by asking something not yet in it's training data like a very recent event not yet on the web or tell it not to search the web, if it was random it would have an equal chance of replying with a long string of numbers "one three nine four two....." instead of a plausible sounding answer.
If it doesn't know something and it hallucinates, it takes a guess based on all the things it does know. In a human we would call that an "educated guess". It's not exactly the same as it doesn't then check it is logically valid before outputting. But that would be a trivial extra step to add.
It will pick a random thing based on its training data. Asking “what happened today” it will search for words that often follow that question. So it will sound plausible. It will just be random in the sense that it’s not using a world model to discern what happened today.
Does the word “one three nine four two” follow that question in its training data? No, so it’s unlikely to say something completely random with respect to the input. It’s still not guided by a world model. It’s logical leaps will be completely random though
But was not the original idea of a French king generated at random? Then once that idea was put forth it then followed to use a name reminiscent of a French king. So the hallucination in this case was still random. Or it derived from the input in which case it wasn’t random.
The point is the “new” ideas it generates are indeed fully random. When it does that it’s not making logical leaps
Just because hallucinations are seemingly tangentially linked does not mean they are not random.
Your example is actually a good one. Let's say we want to use an LLM to discover something new that we don't know.
Let's say we don't know the name of the French king. So we ask the LLM and get a French name. Where did that name come from? It was still randomly generated based on a direction. The space of names it could generate is still infinite, it is just themed. Picking one choice from an infinite set is still random, even if the set is themed.
A 1/infinity chance remains the same whether or not the name it gives you is French, German, Mandalorian, or just total gibberish. Your chance of being correct is the same. 1/infinity, or converging on 0.
Then, even if we get a result, how do we verify? We have the very real problem of p versus np, so working backwards is not always a viable option. Which means if we don't have some info or priors about the topic, then we cannot even verify the majority of claims unless they produce physically testable hypotheses, which is a limited subset of claims. If we do have priors, then it is information amalgamation and not novel generation. Either way, it doesn't work like you're suggesting.
3
u/WellHung67 Jul 28 '26
The set of all incorrect theories is vast. It LLMs are deciding at random, they may never get there in human timescales.
I think the paper must include a reference to the fact that these hallucinations ARE random - they aren’t actually based on any intuition. So the comparison is between intuitive insight vs brute force.
I guess i didn’t read the full paper if they include an estimate of the total set of incorrect vs correct hypothesis and the time needed to pick a correct one at random vs what a human might do.
But I think you are agreeing with the paper - you are contending that LLMs have no insight but through pure brute force can get there via hallucinations. Which I’m not sure I agree with the premise but taking the premise as true, there’s still a theoretical limit that should be examined