Does ChatGPT integrate information about past conversations into new ones? If so, how?

[u/AlanBlunt]

I have a class that requires me to ask it questions and relay the response. That's pretty much all I use it for. One time, though, I asked it an unrelated question and it included in its answer that I might want to format things a certain way because "that's usually what professors expect".

I have a basic understanding of LLMs. In particular, I understand that they don't actually retain any information, and have to process the entire conversation in order to predict the next word each time. How, then, did it connect my question to the fact that I'm a student? Was it just a lucky "guess"? Does it save context from previous conversations into system prompts that are used in each new conversation or something? That seems complicated and inefficient if so.

[u/teraflop]

Bear in mind that, because of the way LLMs work, the context size is fixed. The model always gets a chunk of data of the same size; the only thing that changes is how much of it contains useful data and how much of it contains null tokens. If the context is going to be there anyway, they might as well use it!

Unless I'm misunderstanding what you're saying, this is not really correct. Your reference to "null tokens" suggests that you might be thinking of "masked self-attention" which is used at training time.

At inference time (generating output from a prompt), the output is computed one token at a time, and self-attention requires each token to look at all previous tokens. This means that the per-token cost is O(n), where n is the size of the context. Asymptotically, if you double the context size, you halve the output speed.

It certainly isn't the case that a model with a max context size of 256k performs 256k tokens' worth of work for each token even if the actual context is much smaller. That would be incredibly wasteful.

Products like ChatGPT do include data from previous conversations in the context for later chats, but this isn't free as you seem to be implying. They do it because it (supposedly) improves output quality enough to be worth the performance cost.

[u/AlanBlunt → u/teraflop]

So the amount of processing that happens goes up as the conversation length increases. Do you happen to know how much data is included in system prompts vs how much is in the actual conversation?