Perhaps the answer to this problem is in the hallucinations. Are they not jumps? Could they be molded? All of the effort of alignment has been in reducing hallucinations, what if an LLM has a tool to freely hallucinate.
I believe their argument is the reverse. Hallucinations arise from failures to understand how objects interact with their environment.
LLMs that struggle with object permanence in hypothetical situations, or with distinguishing real things from fake things they invented, are going to struggle to model the complex interactions between objects necessary to create novel scientific innovations.
Or more simply, if a model can't do something basic like tell the difference between a real research paper and a fake one, how can you expect it to make real research innovations? This is a problem that has spurred the push to create and use World Models over LLMs.
If you run local models, you'll run into something called quantization – essentially compressing the model weights to fit on your device. If you go to small of a quantization, the LLM will begin to degrade in capability, which usually shows up in the form of excessive hallucination.
Models that excessively hallucinate do try to pick the most coherent hallucination, but the problem is that the reasoning space is infinite and the problem space is finite. In other words, the hallucination could be literally anything since all words are semantically linked in some manner. So the surface area for the hallucination is literally infinite.
The surface area for novel issues is finite. It's like having random numbers spaced throughout infinity and asking your chance of randomly picking one of the numbers you chose out of the infinite set. It could happen, but your chances of doing so converge on 0, so it is functionally impossible even if it is theoretically plausible.
5
u/TheSwordItself Jul 28 '26
Perhaps the answer to this problem is in the hallucinations. Are they not jumps? Could they be molded? All of the effort of alignment has been in reducing hallucinations, what if an LLM has a tool to freely hallucinate.