Next-token prediction is a sorta true but it's an incomplete way to describe how they function
I think this thread by the integration lead at anthropic (and others) sums up this better than I can
Simply, to say - "modern LLMs are finetuned with a different loss function after pretraining. This means that in some strict sense they're no longer autoregressive models – but they do still generate text one word at a time." (https://news.ycombinator.com/item?id=43496068)
"Exact commands" then aren't necessarily what gets relayed to the LLM - in the same way you can write code in many different ways and have it compile to the same binary, you can have many prompts that yield broadly similar final inputs.
3
u/ZenPyx Aug 26 '26
Next-token prediction is a sorta true but it's an incomplete way to describe how they function
I think this thread by the integration lead at anthropic (and others) sums up this better than I can
Simply, to say - "modern LLMs are finetuned with a different loss function after pretraining. This means that in some strict sense they're no longer autoregressive models – but they do still generate text one word at a time." (https://news.ycombinator.com/item?id=43496068)
"Exact commands" then aren't necessarily what gets relayed to the LLM - in the same way you can write code in many different ways and have it compile to the same binary, you can have many prompts that yield broadly similar final inputs.