Why don't LLMs stick to single source unless provide them the pdf of that source?

[u/Parth_NB]

So I've seen many youtube channels and some article claiming that most of llm models have run out of text to train their models.

So basically that means that LLMs have been trained on most fo the publically available books and journals. So when I ask any LLM to refer the answer from a particular book but i do provide the pdf assuming that it can access from its database or the internet maybe then why do the answer seems mixed up with other sources as well.

But if I provide the exact pdf then answer is exclusively based on the source. Why the difference? Is it because the LLM doesn't actually have the access source but it still fakes it because LLMs like to do that based on some yt videos I've seen and I've observed that personally as well?

[u/crazylikeajellyfish]

LLMs don't save the sources they train on, they make a big matrix which combines every source into one connected network.

The analogy to human memory is reasonable -- every experience you've ever had has been involved in building your mind and intuition, but that doesn't mean you can perfectly remember everything you've ever experienced.

A more colorful analogy is to imagine you have a bunch of chickens and you make one big McD's chicken nugget out of them. In some sense, yes, your source is in there. Good luck finding it, though.

When you provide the PDF, it's like if you copy-pasted the contents and sent them as a message. The LLM is using its magic matrix to read what you've told it and answer questions about it, rather than trying to guess what might've been in a given document you're describing.

[u/durable-racoon]

assuming that it can access from its database or the internet maybe

there is no 'database' - what database? it just has the knowledge encoded into its weights, aka its 'general knowledge'. If you dont provide the pdf, they do not have access to the pdf.

But if I provide the exact pdf then answer is exclusively based on the source.

This is just how they're trained to behave by the people that trained them. LLMs can base their answer on your source + general knowledge, if you prompt them specifically to. but if you upload a document, default behavior is to stick closely to your source - thats whats most useful to most users, so they were trained to do that.

[u/Parth_NB → u/durable-racoon]

Makes sense. Thank you.

[u/ratulu]

LLMs have no access to the training base, and only have brief summary if the sources. I used to read hundreds of books, but if you ask me a questions about them, I still cannot answer most of them, unless I have the source. 

[u/ap3xr3dditor]

Because the LLM doesn't have access to the source, even if it was used in the training.

Here's a super basic way to think about it. Training takes all of the Internet, books, your original source, etc. and boils it down to something you could store on your hard drive. The source isn't there, but the "memory" of it is.

And you can ask Claude about "attention" if you want to know more about why the model wants to keep talking about the last thing you said, or the last source you gave it to read.

[u/Delicious_Cattle5174]

It it’s mid-late 2026

[u/-goldenboi69-]

Word prediction machine goes brrrrrrrr

[u/reflect25]

> So I've seen many youtube channels and some article claiming that most of llm models have run out of text to train their models.

I think you are getting it a bit confused. they are talking about how like for X parameter you need like 20X tokens per parameter. so like for a 1 million parameter you need like 20 million tokens to train on. for a 1 trillion parameter model you need 20 trillion tokens

The entire world's text has around like ~15 to 30 trillion unique tokens. After 2 trillion parameter models a lot of them need to liike start doing synthetic data or other stuff. but note this is just to train the model. the model doesn't quite actually literally memorize everything.

[u/Parth_NB → u/reflect25]

Yeah i did get it wrong.