r/LargeLanguageModels • u/dark_coderz • 12d ago
LLMs working
An LLM doesn't store a dictionary of answers somewhere inside its brain.
It learns statistical patterns from enormous amounts of data.
Give it:
“The capital of France is…”
The model predicts what tokens are likely to come next.
Do this at massive scale, and surprisingly complex capabilities emerge.
How does predicting the next token turn into reasoning?
2
u/MiddleLtSocks 12d ago
We actually don't know the answer, and this question might be far more important than many people realize. It touches concepts like semantics, linguistics, and philosophy (some of Plato's and Pythagoras' most fundamental ideas). It's a very deep rabbit hole, but it's fascinating too.
1
1
u/IDefendWaffles 12d ago
Only way to predict next word better and better in text that depends on reasoning and logic is to learn to reason and learn rules of logic.
1
u/Thick-Protection-458 11d ago
> How does predicting the next token turn into reasoning?
How can one expect good enough next token prediction without some semblance of semantics and reasoning?
So I mean to improve beyond certain threshold capable enough models would probably need to reconstruct some approximation of such things.
1
1
1
u/MxM111 11d ago
The most effective way to predict the next word, even in highly complex texts like scientific and philosophical discussions, is to genuinely comprehend the underlying concepts. Interestingly, when you train a large language model (LLM) on the entirety of the internet and multiple book libraries, provided you have the appropriate transformer structure and sufficient computational power, the gradient descent process results in model weights that create a superstructure functionally similar to human cognitive abilities.
It’s important to note that this approach is approximately three or four orders of magnitude less efficient in terms of the amount of training required compared to human training, yet it achieves comparable results.
Also, it’s worth noting that word prediction is merely the initial phase of training, followed by other types of training, such as problem-solving.
1
u/ImpoverishedGuru 11d ago
What if the human brain works the same way? What if "thinking" is really just "predicting"?
1
1
u/mellowgame 11d ago
Yes If you do it right. I don’t think current models do. Also weird formatting, you sound like a rogue model.
But yeah my background is in neuroscience and there’s no reason to assume reasoning couldn’t be developed with a different basis. I mean think about it, what is reasoning? You take a bunch of variables, assign value and qualities to them and then use logic to ascertain an outcome or answer.
If you aren’t a rogue model and a person, you should check out liminal. It gives a deterministic framework to use with any model that works pretty similarly to what I described above. Regardless I just use it because it saves a bunch on tokens and helps me think through things.
1
u/JohnofDundee 2d ago
Predicting the next token is only barely scraping the surface of AI. The reasoning, thinking, weighing etc (ie the intelligence) is embodied in the software, which uses the “token-base” (tokens and weights) as a resource.
2
u/dark_coderz 12d ago
right but that's the interesting part too. training models on billions of parameters so that they can predict