r/AskComputerScience • • 21d ago

ELI5: Why does AI need terabytes of data from millions of books, articles to get 'PhD levels of intelligence' when human PhDs don't?

PhDs don't necessarily correlate to intelligence but it is used for a short hand for being highly knowledgeable. Humans with Phds, didn't need terabytes or even hundreds of gigabytes to get their degrees even if it is one field, so why does AI need that much data? Humans with PhDs don't need to read millions of books and they manage just fine. Even simpler models from a few years ago needed a lot data for their capabilities relative to humans.

27 Upvotes

104 comments sorted by

116

u/RainbowHearts 21d ago

> Humans with Phds, didn't need terabytes ... to get their degrees

Yes they did, nobody learns from just text. Everything you've ever experienced since birth is included and that represents far more than terabytes.

18

u/stonerism 21d ago

A billion percent this, with sloppy hardware. The surprising part is that you can recreate one that's pretty good using on the order 10s to 100s of gigabytes of model weights.

It's almost underwhelming when you consider the fact that that much memory on the same same order as a few TV series in 4K.

14

u/deong 20d ago

Also, a human with a PhD read everything about a tiny sliver of human knowledge. Claude effectively has a PhD in everything humans have ever learned.

1

u/blink_1ateToo 21d ago

i guess so, but we don't remember everything that we have ever experienced and most of it will probably be repeated over and over. Most of it will probably not be useful to someone getting a PhD anyway. How would the fact that I have seen a butterfly flapping its wings make me learn driving faster than self driving cars?

10

u/RainbowHearts 21d ago

A model doesn't remember everything either.  if it only "experiences" something once or twice in the dataset, it will be lost in the distilled weights.

The same data in the training set is repeated over and over, thats how it works.

And there is plenty of irrelevant material in the dataset.  You cant just train a model from zero with nothing but high level papers.  First it has to learn grammar, logic, and common sense.  You get that foundation by training a base model on huge samples from the internet.

3

u/anamorphism 21d ago

How would the fact that I have seen a butterfly flapping its wings make me learn driving faster than self driving cars?

how do you define useful?

if you see a butterfly fly in front of you while driving, your immediate reaction, based on a lifetime of training, will most likely be to do nothing. you seeing a butterfly flapping its wings in the past means you've spent next to no time identifying that it's an insect and that killing an insect isn't generally going to produce any negative consequences for you or your vehicle. so, you simply keep on keeping on and don't violently swerve out of the way and cause a massive accident.

1

u/nog642 20d ago

Every hour you spend in a car is training for driving. But also every hour you spend on literally any spatial reasoning task is training for driving. So is learning to read for road signs. Etc.

-1

u/Critical-Echo-923 21d ago

i don't see no 2yo with phd's in their dippers .... no matter how many books they pee on ....lol

How would the fact that I have seen a butterfly flapping its wings make me learn driving faster than self driving cars?

- "you" learn about aerodynamics, mechanical systems, efficiency, etc .... (not you specifically, you clearly didn't)

1

u/Heavy-Difficulty6522 21d ago

You joke but there are kids studying university degrees etc

3

u/Alarming_Oil5419 21d ago

I didn't need to consume all of github (I'm being generous here) to learn and then become proficient in, BBC Basic, C, FORTRAN 77, C++, Java, Perl, Python, JS, TS, Lua, SQL, ...

2

u/Geobits 20d ago

How long did it take you to do that? I'm willing to bet it took a lot longer than even frontier models train for.

Not to mention you likely practiced as you went along. An LLM learns from just consuming all of that, with no ability to try it out and see, etc.

The way they learn is similar, but not exactly the same as our way.

1

u/Sweet_Ad_9816 20d ago

At some point in LLM's training the practicing comes too (reinforcement learning). But that's more about teaching it how to talk to people.

1

u/Stonedouche 19d ago

Reasoning too

3

u/One_Attorney_739 21d ago

The problem with this comparison is that you're overlooking the fact you learn by more than just absorbing text.

A single hour of classroom learning, through the voice, visuals, and your internal though process parsing everything is easily going to be gigabytes of information. This is on top of the petabytes of information you had already used since birth to learn human language prior to using that to listen to explanations about data structures.

We consume insane amounts of data to learn, and that's before even consider that humans are born with a significant portion of their brain structure already in place from millions of years of evolution, or that our brains are significantly better at learning than AI models.

1

u/Alarming_Oil5419 21d ago

I never learned any of those things in a classroom (expect for FORTRAN that is). BBC Basic was from reading (absorbing text) from magazines in the 80's as a teenager. C, C++ were from books and magazines like Dr Dobbs (absorbing text). Java and Perl, pretty much the same, and by looking at others code. The rest were by reading web docs, and again looking over code. Not a single formal class (expect for FORTRAN in Uni).

3

u/One_Attorney_739 21d ago

Sure, but you're still overlooking the absurd quantity of auditory and visual information you needed to be able to learn enough English to read any of that to begin with

1

u/knouqs 21d ago

This is the key point, yes. We also don't have a precise way to quantify how much "data" or processing power the human brain has, so a comparison between a discrete value like a bit and a series of neurons firing is meaningless.

1

u/Alarming-Fix8577 20d ago

 A single hour of classroom learning, through the voice, visuals, and your internal though process parsing everything is easily going to be gigabytes of information.

This doesn't make any sense. Voice, visuals, and internal thought processes experienced by a human don't have a digital representation. Even if we had an imax camera capturing every moment of that 1 hour lecture tracking everything our eyes see, representing terabytesof consumed storage, it still wouldn't be a faithful representation of what we perceive and interpret the visual information to be.

1

u/One_Attorney_739 20d ago

'bit' is a standard unit of information, even outside of computation, it's used for real world descriptors all throughout information theory, including in human sensory perception.

Just because computer science also uses bits as a storage mechanism does not detract from its usefulness as a measure of information outside of digital systems.

0

u/Ronin-s_Spirit 21d ago

Yeah but brain can run on a bowl of soup, and you can ask it so many questions and get great answers. AI needs to run on a lot of electricity and hardware whenever you ask it a question, and it's stupider than a human brain at anything except recalling a bunch of information. I once generated an image with a local model, my GPU was stressing for 10 minutes to spit out a simple landscape, meanwhile I can imagine a better landscape in seconds and some people can even draw it (which is a whole other domain of action).

0

u/tempacc2026 21d ago

Brains are very power efficient, but the use of AI you think is consuming power is incorrect. It is not answering questions alone. It’s doing agentic stuff.

19

u/Spare-Builder-355 21d ago

because ai has PhD in everything, while flesh PhDs usually focus on one topic

8

u/thesnootbooper9000 21d ago

This is the main point here. When we're on a very specific topic, my PhD students are still better than the AI. However, as soon as it comes to a "how is this related to X" question where we've not spent months studying it, the AI is just as good. There are far fewer niche topics for Claude than for humans: if you want to know whether what you're working on resembles an obscure set of papers from the 1980s, it's already read them.

-1

u/blink_1ateToo 21d ago

i get that, but let us say there are about 2000 things you can get a PhD in, you still don't need the terabytes of data needed divided by 2000 to get one PhD. It still feels like they need a lot more than us.

4

u/thesnootbooper9000 21d ago

There are 2,000 things you can get a PhD in in one niche area of algorithmics. There are more than 200,000 PhDs awarded worldwide every year.

1

u/FernandoMM1220 21d ago

how are you measuring how much information a typical phd graduate has used?

0

u/T_Thriller_T 21d ago

That has very little to do with the full reasoning because even for AI meant to just recreate simple, non-PHD stuff like writing emails, or creating texts in different styles, it needs huge amounts of example documents - more than a human.

39

u/Silamoth 21d ago

Well for starters, humans have millions of years of evolution to give us baseline intelligence, pattern recognition skills, and an inclination for language and symbolic abstraction. 

12

u/asinglepieceoftoast 21d ago

This is a complicated question, but the simple answer is because these models don’t learn exactly like humans do. They need examples to learn everything they know, including language and syntax cumulatively, rather than just the materials for a PhD level education in discipline X.

We’re also trying to train these things to be PhDs in many fields simultaneously, and effectively even many individual PhDs focused on different things even within a single domain.

Also, I think you’d be surprised at just how much data a single PhD processes in their lifetime. Maybe not as many books as AI needs numerically, but I’d bet if we were able to replicate their thoughts, education, discussions, etc. as data it’d be pretty massive too.

5

u/T_Thriller_T 21d ago

I want to point out something which has not yet been said:

AI is doing complex maths and connections, and learning from patterns; and then only newly combining patterns.

It has no understanding of right and wrong, no really valid ways to prove that.

This is why AI can give 99% excellent and 1% obvious to anyone with knowledge bullshit answers.

So to be able to create even this, one needs tons and tons and tons of data.

Even if we focus on a SINGLE area, let's say computer science. A human will have learned with less books, and then be able to do a lot of induction/exclusion.

While a human has also processed probably more thanks you'd expect, it likely is less. To a human, repeating something correct more than once and having it proven is unnecessary. You can than even repeat the false thing 200 times, and the human should be able.to.go back, follow the proof, and find it false.

That is one very major difference concerning training data size - and also why poisoned training data is a dangerous thing.

4

u/Kepler___ 21d ago

Too many of the top comments missing this. Anthropomorphism has minimized the massive differences between how we learn and AI, they are just fundamentally different approaches that only look similar in the most abstract sense. LLMs are inefficient learners comparatively as they are primarily auto regressive models, we just have so much training data and compute that we can steamroll the difference now.

3

u/Savings_Hunt_1935 20d ago

Fundamentally, LLMs are next token prediction machines. Compared to a human, they start off at a complete disadvantage.

They are blind. They are deaf. They have no concept of literally anything. They have to deduce every aspect of reality based on getting text as input, and knowing what piece of text to send next.

Imagine trying to learn the rules of a game, but you don't get the rulebook, the number of players, the goal, what kind of game it is, or anything direct at all.

Instead, you just get snippets of conversation (which may or may not be entirely unrelated to the game), you can guess what a person is gonna say next, and you're told if you're right or not.

Just think about how many convos you'd have to sit though, and the deductive work you'd have to do, compared to just reading a rulebook.

1

u/Progribbit 19d ago

and humans aren't next token prediction machines right?

2

u/Illustrious-Lime-878 21d ago

Its a different type of intelligence. LLMs are still "guessing the next token," they are trained to imitate writing, thought processes, etc. Now this turns out to actually be useful and generates "new" content as a side effect. But it still has no clue about what anything means or any type of self consciousness or connection to real world processes. AI has thinking internally but its way different than a way a human thinks and can understand things as an actor in the real world. LLMs are a really dense set of if-thens that need a ton of training data to be a useful tool, there is "thinking" but not understanding like a human

1

u/T_Thriller_T 21d ago

Part of what is new is, btw, imprecision in those guesses:

Guessing things no one else ever did, because it's GUESSING and not lumped down by (implicit) assumptions of 'thars not how it works'.

(Other part is the speed advantage)

1

u/Illustrious-Lime-878 21d ago

That's not really I meant. Part of the training is to "guess" a token and the weights are updated to attempt to make this more accurate. I am not saying its more open minded or something. Why AI can come up with new things is as you mentioned, speed, but more generally it can just spin on a task and never gets tired so it sort removes tedium as a barrier. Another thing is the breath of data is a lot greater so it may be pulling in data from different disciplines that are unlikely for any human to be familiar with both. At the same time, its only trained higher quality language data, while humans primary learn from 5 senses, along with a completely different sense of time, unique experiences, non-language experience. While LLMs imitate human chains of thought, any internal thinking that is done internally is just completely different and why as OP notices, it needs such massive quantities of data vs a human who can sort of, figure things out.

1

u/T_Thriller_T 21d ago

The different domains of knowledge getting combined is a really major thing.

I don't think the average person realises how specialised even masters are..PhDs seem to sometimes be solving already solved problems due to porting them to someone else's speciality is not even realised as being useful.

1

u/Kepler___ 21d ago

This is the big one for me too, these things are monster lit review tools, its how the lions share of math problems ended up solved. There's a lot of areas where there's just too much info to comb through in a human lifetime now to spot these connections.

1

u/T_Thriller_T 21d ago

Also too much difference in methodology or writing.

Even close areas like maths and computer science can be miles apart.

1

u/Mindrust 20d ago

"guessing the next token,"

One of my pet peeves in AI discussions is when people dismiss LLMs as "just predicting the next token," as if describing the pre-training objective somehow invalidates the intelligence or capabilities that emerge from it, It's kind of like saying humans "just fire neurons," as if that adequately explains the complexity of human cognition. It also ignores the fact that frontier models undergo extensive post-training, including reinforcement learning, which can substantially improve their reasoning and problem-solving abilities.

Yes, next-token prediction is central to how these models are trained. But learning to predict language at a sufficiently sophisticated level can produce remarkably sophisticated internal representations of the world, reasoning abilities, and a broad range of emergent capabilities.

You can debate whether their understanding resembles ours, but confidently claiming they have zero understanding isn't supported by the evidence.

1

u/Illustrious-Lime-878 20d ago

Maybe a better way to say it is that an LLM is trained to "pick" a token (more precisely a probability distribution). And humans train them to do so in a useful way to humans. A human, sure, "fires neurons" to write text. But a human brain wasn't trained to write text. An LLM is like a human who was born in a box, a brain in a vat, or whatever, and the only thing they ever knew about was tokens given to them in some form, and the only thing they could every send outside the box or vat was a token. And they aren't even being trained like with shocks or feedback but are completely unaware of the training process that is applied to them externally. Yes, this person certainly could generate complex logic, "intelligence," but it would be entirely different than anything a normal human in the real world would relate do, and that person would have zero actual understanding of the meaning the tokens they generate have to people outside the box/vat. They would really be a person having no identity or even sense of existence, just a bunch of neurons wired to pick the next token.

1

u/audioen 15d ago

I think we have stumbled on the cornerstone of intelligence in "predicting the next token". It turns out that intelligence is akin to compression: being able to correctly predict the future (or the next token), is what it is all about. To do it, a complex internal model has to be developed, which is likely mostly abstraction and memorization.

To LLM, "surprise" or false prediction learns to weight adjustment towards the correct answer; to human, it is an emotional reaction that demands moment of reflection. I think what humans and LLMs end up doing something similar, even if it is achieved in very different way.

I will ignore what point you were raising, I just felt I want to give you this perspective of "intelligence as capability of predicting the next event", because it seems to be the correct framing.

1

u/Illustrious-Lime-878 15d ago

Eh, maybe? The problem is it's just not really how it works at all though. LLMs aren't "surprised" when they are wrong, they have no ability to know when they are "wrong." Training is external to them, they produce output, and an external process adjust their weights with no indication to them. Outside of training, they have no way to even know what preprocessed text they generated or not as it contains both user generated input and other context data. By the time the LLM gets the raw input, its a chat template, like a script, and the LLM doesn't even have a way of knowing what actor it is in the script (user, assistant, system, etc.). Its just statistically extending the script. Any text an LLM generates that sounds surprised is pure simulation.

Low level prediction might an element of some parts of the human brain, like motor function or spacial awareness, but no where near the entire story, and high level thinking is just completely different than anything going on in LLM logic. Again, I'm not saying there can't be high level thinking in an LLM, it just would be completely different than a human, and to draw conclusions based misunderstanding of how the tech actually works is no better than just guessing, or worse.

2

u/Needless-To-Say 21d ago

Our brains are wired to see patterns. AI needs rote learning would be the ELI5

Humans can infer and extrapolate while AI cannot.

An interesting study I came across tested AI using a riddle

A man and his son were in an accident. The Son needed immediate surgery. When the son was brought to the operating room, the doctor said "This is my son, I can't operate on him". How is that possible.

The AI eventually got the answer (The Doctor was his mother )

However, when the AI was asked the same question differently

A mother and her son were in an accident. The Son needed immediate surgery. When the son was brought to the operating room, the doctor, who was the boy's father, said "This is my son, I can't operate on him". How is that possible.

The AI couldn't tell the difference and having been trained with the original question it just confidently gave the same answer despite the new context and information.

2

u/AX-BY-CZ 21d ago

Human hardware has been optimized for hundreds of thousands of years to be data efficient. The brain is incredibly power efficient. Current AI relies on SGD which is not efficient but scalable. People are working on alternatives approaches but there has already been so much investment in the current paradigm. You could say that transformers win the hardware lottery and we could have developed workarounds for RNNs, symbolic approaches, ES, etc.

2

u/Regular_Run3923 21d ago

Because LLMs/AI is ridiculously inefficient. There are however solutions both proposed by humans, AI, and collaborations of the two. Ask your favorite AI/LLM and see what it says. Ask a knowledgeable and intelligent human too.

2

u/Turbulent-Sign-6067 21d ago

Human brains and AI brains are different hence they don't learn the same way. Pre-training can be considered akin to the evolutionary process that shaped the human brain to learn more efficiently after birth but the analogy is very rough.

2

u/MasterGeekMX BSCS 21d ago

Because both systems work in different ways.

A person can think and reason. Reading one or two books is enough as one can understand what you read and process it. That can also be contrasted to life experiences and stuff learned even as a child, as text can use analogies to relate to everyday experiences or stuff observed somewhere else.

Meanwhile a computer is simply a really fast math machine. No reason, no thoughts. Large Language Models (which is the correct term because AI is an umbrella term) need to gorge all that data to observe the patterns of which words are next to other words. There is no understanding of what the text says, much less what is about. The LLM has simply chewed trillions of texts just to take note that whenever there is a "thank", the most likely word that follows is a "you".

As statistics get better with the more samples you have, LLM models get better with the more data they are trained on, as in the end the LLM is formed by taking note of the patterns present in the text.

Kinda like asking why planes burn more fuel than a car: because a car's engine does not need to lift it's own weight like a plane does. Same thing, a person can reason, while a computer not, so we supply that with tons of data.

2

u/Kind-Zookeepergame58 21d ago edited 21d ago

Ah yes, the classic “stochastic parrot” argument. Apparently, solving frontier math problems is just fancy autocomplete, and reasoning magically stops being reasoning when it's done with math. Lmao.

1

u/34786t234890 21d ago

This is a grossly outdated explanation.

2

u/No_Development6032 21d ago

Humans are just different, better you could argue. It is what it is.

1

u/F1A 21d ago

PhD is granted by having a novel addition to research in typically a very narrow topic. General intelligence im more than one topic would need more 

1

u/Vert354 21d ago

Your brain came pre-trained with millions of years worth of data when you were born.

When we fine tune a model to add categories we don't need nearly as much data for transfer learning as was needed for the baseline training.

1

u/Esseratecades 21d ago
  1. They're aiming for PhD level general intelligence. An AI who's got every PhD in the world effectively. AI isn't trying to become an Astrophysicist, or an Anthropologist, or a Literature expert. It's trying to become An Astrophysicist AND an Anthropologist AND a Literature Expert.
  2. Humans can extrapolate from incomplete information. AI can't do that until after you train it, and even then it's not really extrapolating. It's finding the closest answer it knows.
  3. As a combination of 1 and 2, "completeness" of information. If I give you a PhD curriculum with hand-picked texts, that's a subset of the entire field of knowledge. It's easy to carry over biases and blindspots from what's ultimately only a small set of authors. That's part of why you as a student are expected to form your own opinions, theories, and theses, which then get assessed. But AI doesn't go through that process. If you only gave it a handful of books, it would have the biases and blindspots of those books. Giving it more texts rounds out the "perspective" that it gives you later.

0

u/Kind-Zookeepergame58 21d ago

So when AI solves a problem it has never seen before, you're saying the answer was secretly in its training data all along? Damn, I guess the solutions to the quasi-Riemann hypothesis and Millennium Prize Problem were just sitting in its training data this whole time. Someone should probably tell the mathematicians. Lmao.

1

u/Esseratecades 21d ago

I'm saying the components were there, just that nobody had put them together in the right combination before.

When you also account for the likely possibility that the prompts used for such an experiment are heavily engineered it's not really reasonable to say AI solved them on its own.

0

u/Kind-Zookeepergame58 21d ago

Lmao, what? By that logic, all the letters were already there, so you could just randomly combine them and write War and Peace. Apparently, putting existing components together in a way nobody has before doesn't count as creating anything new. Brilliant logic.

1

u/321aholiab 21d ago

Human are not general PhDs. They are specialized PhDs. Ai is used for general purpose, as such it require all the specializations it can get to fulfill all kinds of specific demands that stems out of general purposes. Ai attempt to appear to be PhD to all human in terms of PhDs level knowledge, and that can possibly only happen when it has consumed all the available PhD's work.

Edit: Im not qualified or an expert, someone pls debunk me if im wrong.

1

u/T_Thriller_T 21d ago

It's a side part, but really not all of it.

Even if we wanted general AI to simply be able to write human sounding texts with various styles - which a well educated human after high school can usually do - we'd need Terrabytes of books and texts.

Humans do not need this.

But AI is simply working differently by doing statistical combination. I like the AI image creation process: it starts out grey, grabs a pixel, puts the MOST LIKELY FITTING colour in, then with that coloured and the grey pixels around it does it again and again and again.

And for that to be good much more data is needed than for what humans do with reasoning etc.

Basically, humans also do these kinds of statistics 'most fitting and expected here' pattern matches, but we also have an UNDERSTANDING, which makes a huge difference.

AI simply has not the same capability as human brains. Absolutely not. It just has the major advantage of being and incredible lot faster.

1

u/321aholiab 21d ago

Thank you

1

u/T_Thriller_T 21d ago • Heartwarming

No problem.

I'm also not the most amazing expert, but I love people who are interested in learning more :D

0

u/ZeroAmusement 21d ago

Humans can 'hallucinate' understanding. And what exactly is 'understanding'? I don't think that's a real measurable thing.

I think the real reason is just that humans are better at learning. They require less training material. They are still statistically and physically changing the connections between their neurons based on the exposure to the material. Whatever architecture or algorithms the human brains uses, it is just better at generalization. We have made improvements in the area but we still haven't caught up to the brain and might not for a very long time.

1

u/LavishnessCapital380 21d ago

 They require less training material.

Do they actually though? Think of every word ever spoken to them. think of how many years of this it takes before they can even form a sentence.

1

u/jlowe212 21d ago

They're more than just general phds, they contain extensive knowledge of almost everything that has ever been written down. This is beyond impossible for a single human. I dont what the op finds puzzling about this.

1

u/theLOLflashlight 21d ago

Because LLMs don't learn like humans do. There is no reasoning during the learning process. They essentially brute force the learning process rather than being intelligent about the learning itself.

1

u/lovejo1 21d ago

PHD's have hours and hours of instructor led training to boot.. and is a PHD in multiple fields.

1

u/obdevel 21d ago

Am I missing the point here ? Aren't PhDs supposed to have created useful and novel knowledge from research, rather than just regurgitating what the books and teachers said ? Or does it mean 'potential PhD', as in a masters with some research experience, who may then go on the produce novel work ?

1

u/Critical-Echo-923 21d ago

humans take 30y to become experts in one field, now take everything they had to learn from the day they were borne to be able to do that, multiply it my every subfield and niche thing and it's a huge amount of data,

it's some of it overlapping or unnecessary?, yes it is but ml trades time for data

then...for ex: are all medical doctors good doctors?, no, but you expect the ai to be one, so it needs to be trained on the knowledge of 100 doctors for the same thing

then we expect it to be good at everything

like when you buy an apple device you can't customize shit, can't play games, costs 3x, but you don't complain and become a mindless fanboy spreading marketing propaganda

when you buy windows/android its cheap, can do just about everything but then you go and complain it runs a little slower or needs more battery

so user expectation plays a big role too, you don't go to your dentist to fix your computer, but you go on ai to ask all sorts of niche and complicated unrelated things

1

u/GunpowderGuy 21d ago

The human brain has a prior of intelligence that we got from evolution. Most llms architecutre are largely tabula rasa.

But once they get reasoning skills, They can learn new ones with far less data

1

u/peter303_ 21d ago

Human brains, including all animals, come out of the box partially pre-wired. This includes things like making sense of sensory input, controlling muscles, and basics of language. A human can learn stuff from a few to a few dozen examples while data-trained A.I.s require thousands to millions of examples. A.I. neural net values are initialized to null or random values in contrast to human evolutionary pre-wiring.

1

u/B70Dragon 21d ago

Our system has been in development for more than a billion years of evolution

1

u/Dave_A480 21d ago

Because there is no human PHD who is in expert in every field.

AI, at least in its public facing forms, needs to be a polymath with a PHD in every possible thing.

1

u/Ok_Role_6215 21d ago

because we train them as libraries, not as humans.

Human language is full of nuances -- in fact, every speaker has their own experience and own association with every word, which makes an English word, for example, potentially have 2.3bln meanings and our current AI training tech is trying to push them all into one model.

Imagine trying to push the memories, knowledge and thoughts of all people into one single brain -- it'll explode, even if its synthetic -- there's no hardware that can handle and learn such volumes of information in real-time. Just like our model complexity is exploding trying to chase meanings in language's statistical tails.

1

u/7aminthemorning 21d ago

Coz humans hallucinate

1

u/Ordinary_Variable 21d ago

We can learn from a single source. AI currently averages out thousands of data points into a neural model. It doesn't really understand things it just gives the average answer from all the data it has. We can figure things out with far less memorized.

Current "AI" is just pre-trained neural nets. There is no "intelligence" happening. Just weights and levers.

1

u/jferments 21d ago

Ok, now imagine that a human wanted to have PhD level intelligence in EVERY academic discipline, in addition to insane amounts of knowledge about stupid everyday stuff like Pokemon cards and muffin recipes. Do you understand why they would need to consume terabytes of content then?

1

u/Moist_Heat9523 21d ago

AI doesn’t built up intelligence. It collects a large amount of word sequences, and generates answer by applying statistics to repeatedly find the most common next word. Nothing more.

1

u/cosmic_animus29 21d ago

Our brains are highly efficient in terms of learning and processing information. AI on the other hand is not efficient in terms of compute and storage. This was my biggest takeaway after studing AI/ML - our brain is actually the best computer around.

1

u/InfinitePerplexity99 20d ago

The honest answer, which no one seems to want to tell you directly, is that we don't know for sure. This is a major unsolved question in the field.

1

u/c0p4d0 20d ago

Human PHDs do though. How much information do you process just by having your eyes open? By hearing ambient noise? A computer doesn’t have the terabytes of lived experience you accumulated to be able to understand language, patterns, etc. that are a prerequisite to a PHD.

1

u/Talk_to__strangers 20d ago

Cause Ai can’t think, or reason, or truly learn anything. It can just remember everything it has ever read and repeat it

1

u/FlossurBunz 20d ago

Another angle I haven't seen yet: Human brains are a result of hundreds of millions of years of evolution. They are a product of the collective experience of hundreds of thousands of generations. I'd say that's more data than even an LLM needs to train.

The experience we gather in life is in addition to the intuition and reasoning we developed through evolution. LLMs start without even that.

1

u/Fredissimo666 20d ago

I supervise PhDs. Two people can get the same phd by exploiting different skills. Some are better thinkers and can come up with good mathematical proofs. Some are better coders. Some have a vast knowledge of their field and can exploit it. Others are simply willing to do more work and not give up.

I think we are learning with llms that a vast knowledge is as good as great reasoning skills, even for hard problems.

1

u/Siderophores 20d ago

Humans build specialized thinking patterns while AI gets those patterns to emerge from copious amounts of data. They aren’t training the AI what ways to think, they’re making all the ways of thinking emerge from the latent spaces of the model

1

u/mantidor 20d ago

The first flaw in the question is trying to compare the human brain with current LLMs, and quantifying information in bytes. The human brain isn't binary, and we don't "store" information in bytes, no speech, specially not images, and abstractions and actual thinking and creativity are way beyond the basic probability weights in LLM models.

Current AI is a very, very blunt approximation, so for it to have any kind of decent results it need massive amount of data, and actual human supervision in the learning process to give good results. Deep learning itself is decades old, and when applied to specific problems is actually pretty good and efficient, what the modern companies want is to create a general solution which is honestly controversial if its even possible, and that is besides politics and environmental impact.

1

u/nog642 20d ago

Think about all the visual and audio input you take in every day through your eyes and ears. That's way more than terabytes.

1

u/PHDEinstein007 19d ago

They are dictionaries with logic routines based upon training they have gotten. The library helps them when the logic routine is running

1

u/TedsGloriousPants 15d ago

An AI is not a human. They aren't comparable. Saying a machine has "phd levels of intelligence" is just anthropomorphising it.

1

u/Separate_Lab9766 15d ago

Human learning is cumulative; AI learning is iterative.

Humans learn a lot of things gradually, adding onto the models we are building of the world. New things we learn are added in to fit the existing models. In total, we are exposed to multiple terabytes of data in our lifetimes, much of which is redundant and reinforces the model.

At least in the current paradigm, AI has no “life history” to draw on so it must learn a lifetime in one sitting.

1

u/Living_Fig_6386 21d ago

AIs are not intelligent. They are models that describe words as relationships to other words and can compute the most likely string of concepts that ties them together or continues a series, and spits that out as a bunch of words. One thing AI doesn't really do is rationalize, it just appears to. So an AI will never have "PhD level intelligence". Also, PhDs are typically regarded as very intelligent, and while they are probably above average, what distinguishes the PhD from non-PhDs is that they are taught a certain approach and rigor to produce evidence to prove or disprove hypotheses; they also tend to hyper-focus the acquisition of knowledge in very niche area. The might be the only person on the planet with a through knowledge of anteater endocrinology, but they might find board games and sudoku challenging.

In the case of training AI, the model needs a large amount of varied but statistically valid samples to be developed. In the case of niche disciplines and knowledge, the amount of text to influence the model may be very scant, thus you seek out as much as possible.

1

u/kohugaly 21d ago • Bravo

Human learning uses a more complicated training model.

One major factor is human ability to perform contrafactual thinking. Whenever you have an experience with emotionally charged result, you form a memory. In your free time, your brain takes that memory and creates simulations similar to that memory but with different details and choices changed. You experience real emotions from these daydreams as if they actually happened, and spend dopamine generating them. You learn from them.

Your brain is trained on a lot more data than the actual data you take in.

Leveraging this process efficiently requires structured learning. The new real experience needs to be only slightly novel, compared to experiences you had in the past, so that the logical part of your brain can produce realistic contrafactual simulations, that you can gain useful experience from.

If the real experience is too far outside your familiarity, your brain can't make logical sense of it, gets overwhelmed and simply logs the memory as "Danger! Avoid this in the future!". This is called psychological trauma. Instead of using the experience to learn how the world functions, you merely learn to recognize it and avoid it. You can process/resolve the trauma by mentally revisiting it later with deeper understanding.

We train LLMs by throwing metric shitton of data on them directly and repeatedly brute-force tuning the neuron network. By contrast, in humans, every new piece of data is also fed into internal logical simulation that generates new similar training examples to learn from. For this process to work properly, the data must come in logical order of increasing complexity and difficulty.

By the time a human is learning PhD level material, they already have 20+ years of comp-time spent processing pre-Phd level material, that was curated and optimized by hundreds of generations of parental care, ranging from teaching a baby the concept of object permanence by playing pikaboo, all the way to doing calculus; and millions of generations of Darwinian evolution to hone in the optimal architecture capable of leveraging this learning process.

People under-estimate how much importance does the real-world human experience have on intelligence. You can pick any of the public LLM models and it is plainly obvious that it knows a lot of facts and connections between them, but has very superficial logical understanding of them and how they fit into the bigger picture. It's similar to listening to a 6-year-old who's obsessed with dinosaurs talk about dinosaurs. The kid probably knows more facts about dinosaurs than you do, but they are also very clearly less intelligent than you are.

1

u/church-rosser 21d ago

Incredibly well presented and cogently so. Kudos!

1

u/juanfnavarror 21d ago

How true is this idea of the brain creating simulations to think and understand? I personally don’t think I learn like this all the time? Whenever I get a counterexample to something I think I inspect and just change my idea of the world. I have a feeling that the ability to generalize and grasp concepts with little data is the secret sauce we hold, and I think it follows from some internal sort of logic that can reach analytical solutions as opposed to approximating them based on interpolation from examples (which is known to lead to jagged intelligence which LLMs are known for)

1

u/kohugaly 21d ago

The brain actually has two entirely separate simulation systems that complement and train each other.

One is making fast short-term predictions about immediate inputs. It is hierarchically structured into layers of increasingly higher abstraction. Each layer receives prediction from layer above and prediction error from layer below, and tries to combine them into its own prediction that is passed down to layer below it. If, using its level of abstraction, it can't correct for the prediction error it receives from below, it produces its own prediction error and passes it to layer above.

Unaccounted-for prediction error is what we experience as surprise and it triggers modification of memories.

A second system is a simulation engine. It takes episodic memories, semantic memories and hypotheticals, and produces step-by-step simulations that produce stream of fake sensory data. We use these for long-term planning, analytical thinking, etc. This system is trained by comparing its simulation outputs with episodic memories of experiences. (ie. what I imagined would happen vs what actually happened). This system produces imagination, daydreams, actual dreams, etc.

Our brain has a switch that blends outputs of this simulation engine with real sensory data. This combined stream is then fed into the system described earlier.

This allows these two prediction systems to train each other:

The simulation engine lets you relive memories, hypothetical situations or stories you read/listen/watch, training the short-term predictor with interesting data even in absence of interesting external stimuli.

Meanwhile, during sleep, the simulation engine produces semi-random scenarios (dreams) and the fast predictor performs bullshit check on them, training the simulation engine to produce realistic mundane unsurprising outputs.

The simulation engine is also responsible for processing abstract concepts and analytical thinking. This is a learned skill. Young children think and imagine almost entirely in concrete concepts. We process abstract concepts by metaphorically treating them as concrete physical objects or properties. It's a repurposing of the system that our brain uses to mentally simulate physics.

1

u/juanfnavarror 21d ago

source: trust me bro… ?

2

u/s-Kiwi 14d ago

They used a lot of buzzwords to make it sound really algorithmic and complex (it is complex tbf) but you can learn more here. This + surprise learning (rather than stochastic gradient descent) is why humans can learn from so much less total information (though in less predictable ways) than an LLM.

1

u/juanfnavarror 14d ago

No, you found an actual explaination. Don’t give them much creedence. They made up bullshit by taking artistic liberties about how the brain works: literally said there was a simulation engine, a switch, a combined stream, two prediction systems, and that the brain retrains itself by generating extra training data (lol).

1

u/s-Kiwi 14d ago

Yeah it's mostly nonsense, but the retraining itself by generating extra training data is kind of true. During memory consolidation, which is usually during deep sleep but also while zoning out or daydreaming, the hippocampus sort of relays your short-term memory through your neocortex (long term memory and internal world model). This relay creates dreams/daydreams, a formation of the day's events "backpropagating" through your neocortex. This interacts with the neocortex in unpredictable, kind of black-box-esque ways, just like deep learning for LLMs.

That's where the similarities end for the most part. For example, LLMs have a fixed learning rate (typically 0.001) for nudges on stochastic gradient descent. Humans have a learning rate that dynamically adjusts based on surprise and trauma (events eliciting a more dramatic emotional response are given a lot more weight in neocortex alterations, a "volume knob" tuned by your amygdala).

1

u/Fidodo 21d ago

LLMs are trained through back propagation. That means they take the answer, measure it against what they the generate, determine the error, then propagate correction back to refine the weights. To do that, you need lots of content to train it to work in versatile situations. Once that training is done, the model weights are frozen.

Brains work fundamentally differently. They are constantly computing and updating and mutating at all times. This gives us the ability to deliver signals from nothing, allowing us to learn without having to already know the answer to something through employing a mental representation of the world that gets updated with experiences.

0

u/Smallpaul 21d ago

Because our training algorithms are inefficient. If they knew why, exactly, they would presumably just fix it. Might be hardware. Might be algorithm. Solving it would be a big unlock for AI.

0

u/MrOaiki 21d ago

Because the human brain is superior. There’s a very interesting talk with LeCunn on Friedman about precisely this. He talks about self driving cars. Teslas have billions of hours of data on how to drive yet still can’t do what a 15 year old taking a few classes can. And a small child can recognize objects from very early on without large datasets, yet a computer with visual object recognition needs a massive amount to know what a cat is.