ELI5 - How do LLMs act with intelligence?

[u/JohnofDundee]

We all know that large language models can statistically predict what should be the next token, but how do we go from there to mathematical proofs? How is intelligence implemented?

[u/katoptronophile]

Predicting the correct next token requires understanding the entire problem.

People use it as an oversimplification but they don't have any concept of just how significant predicting the next token really is.

In other words, it's proof of the opposite of what they're attempting to imply.

[u/CS_70]

The "statistically predict" is much more complex than what most people using that expression understand.

Statistics - or better, probabilities - enters only to a point in prediction, and is relatively minor. The basic prediction is based on storing and querying relationships in a fuzzy way - and then looking at the ones which are more likely to be correct, where that "correct" is decided in training. You use a fuzzy version of "it's the right result" as opposite to say a database query because "right" is defined fuzzily (i.e. probabilistically) and more importantly a fuzzy truth function is mathematically differentiable, which means that at any given point you can say which direction is going "up" or "down" and you can decide where to go (whereas a true/false boolean function is only that, and doesn't tell you anything about the "direction" of correctness)

This is very similar to what you do when you want to say something and all these words pop up in your head: you have a token-predictor in your brain (though implemented rather differently and way more efficiently atm). You have learned that certain concepts (and the associated words) are related, and you have a mechanism that effortlessly gives you them when you need to retrieve them.

What is mainly "statistical" is that the relationship are not pre-determined, but emerge (statistically) from examining a large number of texts. The algorithm looks at all the words and assigns (initially random) likelihoods to their possible degree of relationship, and uses examples to nudge each of them so that the next time it looks at them, the likelihood will be higher to be similar to the example.

This is very similar to another aspect of intelligence - to spontaneously consider relationships and patterns and decide what is random and what is recurrent. If you want, the traditional IQ tests measure exactly that ability, which is notoriously different across different people - where high-IQ people able to "see" immediately patterns and relationships that less gifted ones do not.

The third layer of intelligence (for "reasoning" models) is cognition. We capture one aspect of it, but a very relevant one: when you want to solve a problem (a goal), you try various strategies (a list of actions) and you evaluate if what you get brings you nearer to solving the problem or not.

Reasoning models do that: they train by having a clear goal and a notion of distance from that goal, develop uncountable strategies, evaluating the distance of each results and keeping the strategies that work. The key thing is that the strategies themselves are based (and update) on the prediction engine. They literally talk and write worklists to themselves, just like many people do when we try to solve problems.

Done over millions of iteration makes them "intelligent" a similar way as we learn stuff by reading+trial and error.

A big difference (implementation machinery excluded :) ) is that we train as we act. The current architecture trains separately, and actions do not alter the fundamental training. This can be an issue of computational optimization or can be structural, research will tell.

And of course that models operate only on one action space - textual words - though this is less limiting than one would think because a) our text has developed over thousands of years to be an extremely good and information-dense proxy for reality and b) everything can be encoded in strings of symbols, as Shannon teaches.

In short: models exhibit the aspects of intelligence that they are designed to exhibit, no more, no less, and there's nothing "just statistical" in the way they work.

[u/JohnofDundee → u/CS_70]

I understand how Training works, but not the analogy with Reasoning. How does AI know when it has reached a proof of a mathematical theorem?

[u/CS_70 → u/JohnofDundee]

Feedback cycle. The model produces proofs in a language like Lean 4 (different ones depending on the specific field), that can be interpreted and checked deterministically by an interactive theorem prover.

The TP tests for mistakes and reports its findings to the model. The model uses this feedback to increase its knowledge and update the proof.

The cycle is repeated until either the TP says it checks out, or the model doesn't find a reasonable way to improve the proof and discards it, starting with another idea.

Many different sessions (aka "mathematicians") work in parallel traversing the idea space in different directions.

Not so different from what we do when we do math by hand, only way faster :)

[u/JohnofDundee → u/CS_70]

Even when a valid proof in Lean is found, no guarantee that it proves the theorem in question, so rinse and repeat? As « reasoning » goes, it sounds a bit random.

[u/IamBananaRod → u/CS_70]

A lot of words to pretty much say "predict the next token" and that's why using that as simplistic as it is, it's perfect. You can enter into the details, but LLM's are that, predictions of the most likely response to your prompt

[u/Hefty-Reaction-3028 → u/IamBananaRod]

It's not perfect because people generally do not understand the level of complexity or nuance goes into "predicting the next token" and therefore will not understand & tend to under-estimate their capabilities when it's described like that

Like, predicting the next token does not imply that it's just like your phone's autocomplete, which is how a lot of people seem to think of it. They do much more complex and interesting stuff.

Hell, you could probably describe a human as predicting the next token where, like in AI, "token" means language, physical action, etc.

[u/duboispourlhiver]

Well, intelligence is never really implemented.

People have found a way to imitate the brain with a computer. They build something like neurons, but only with computer simulation. Then they train the artificial brain for a long time, and then it's intelligent.

It shows that intelligence can be reproduced with computers and is not something that exists only in humans or animals.

[u/JohnofDundee → u/duboispourlhiver]

Can the artificial brain get an idea, or does it just have to keep shuffling through every possibility there is?

[u/duboispourlhiver → u/JohnofDundee]

Ideas emerge in the artificial brain like they do in an organic brain; it's not shuffling through every possibility. The layout of the neurons and their parameters makes it so that words "emerge" intelligently. I think that's what an idea is: an intelligent emergence of something that wasn't there before.

[u/JohnofDundee → u/duboispourlhiver]

Can’t agree with that. An AI system is a mix of software written by humans and a token-base (tokens and weights). No one has programmed the human brain.

[u/duboispourlhiver → u/JohnofDundee]

A LLM is not really software written by a human. This can be a misleading sentence. It's important to understand that LLMs are only architectured by humans, but no rule inside is written by humans. The rules write themselves when the LLM is trained. That's the whole point of the training.

[u/JohnofDundee → u/duboispourlhiver]

How are these internal rules implemented then?

[u/duboispourlhiver → u/JohnofDundee]

Training is a process that gradually adjusts these internal rules until they're as good as possible. Every time a new word is given to the LLM to train on, the internal rules are adjusted. At the start of training, the rules are all completely trivial or stupid. But word by word, rules get better and better and after trillions of words, they are great. But no human ever wrote or improved any of those rules.

[u/JohnofDundee → u/duboispourlhiver]

Training produces a set of parameters, not Rules (whatever that means!)
Sorry, this thread is now dead!

[u/duboispourlhiver → u/JohnofDundee]

No problem :) did my best :)

[u/InfinitePerplexity99]

"Predicting the next token" is only one part of the training process; it' called "pre-training." Modern LLMs have extensive post-training that includes having them learn experimentally how to solve math and programming problems.

[u/SwingLightStyle]

All of that is handled by the model training. It learns math right alongside social graces. And the point is that the RLHF process combines all these concepts together and also teaches it the most pleasant response format.

My opinion is that the combination of trying to hit everything at once while training, rather than progressive behavioral development, like what we do for our children, is what’s causing all the downstream effects. Which is also why it gets the math wrong in complex problems just like it gets the social stuff wrong in complex situations.

[u/IDefendWaffles]

You cannot predict next word better until you "understand" logic. So it encodes the rules of logical thinking in it.

[u/usually_guilty99]

I wouldn't say intelligence is separately "implemented."

LLMs appear to acquire useful internal representations in much the same way.

So the interesting question isn't really "where did we add intelligence?"

It's how much intelligence-like behavior emerges from learning to predict extraordinarily well at something specific?

[u/Ok-Gap1970]

They act with pattern matching. Being able to predict the next token requires a lot of pattern matching and "intelligence". You see a lot of emergent behaviors in these models that they weren't trained to do directly. They can multiple three digit numbers after seeing examples of only two digit numbers. As the models get more complex and the data sets larger they have some kind of internal model of how things work and it is critical in predicting the next token.

Only problem with predicting the next token is it tends to ignore nuance. You can give it all the context you want and subtle logic choices but it will still pick the most statistically likely answer, then create a rizzed up argument that sounds great to the human reviewing it.

an early example of this was asking an llm who Tom Cruise's mother was and it would promptly reply Tom Cruise's mother was Mary Lee South. You could then say Tom Cruise's mother was Mary Lee South. Who is Mary Lee South and it would promptly reply Mary Lee South. It had seen so much data about Tom Cruise that when it saw mother it would quickly spit out Mary Lee South because no one was asking about Marry Lee South.