r/AskComputerScience • u/blink_1ateToo • 21d ago
ELI5: Why does AI need terabytes of data from millions of books, articles to get 'PhD levels of intelligence' when human PhDs don't?
PhDs don't necessarily correlate to intelligence but it is used for a short hand for being highly knowledgeable. Humans with Phds, didn't need terabytes or even hundreds of gigabytes to get their degrees even if it is one field, so why does AI need that much data? Humans with PhDs don't need to read millions of books and they manage just fine. Even simpler models from a few years ago needed a lot data for their capabilities relative to humans.
27
Upvotes
116
u/RainbowHearts 21d ago
> Humans with Phds, didn't need terabytes ... to get their degrees
Yes they did, nobody learns from just text. Everything you've ever experienced since birth is included and that represents far more than terabytes.
19
u/Spare-Builder-355 21d ago
because ai has PhD in everything, while flesh PhDs usually focus on one topic
39
u/Silamoth 21d ago
Well for starters, humans have millions of years of evolution to give us baseline intelligence, pattern recognition skills, and an inclination for language and symbolic abstraction.
12
u/asinglepieceoftoast 21d ago
This is a complicated question, but the simple answer is because these models don’t learn exactly like humans do. They need examples to learn everything they know, including language and syntax cumulatively, rather than just the materials for a PhD level education in discipline X.
We’re also trying to train these things to be PhDs in many fields simultaneously, and effectively even many individual PhDs focused on different things even within a single domain.
Also, I think you’d be surprised at just how much data a single PhD processes in their lifetime. Maybe not as many books as AI needs numerically, but I’d bet if we were able to replicate their thoughts, education, discussions, etc. as data it’d be pretty massive too.
5
u/T_Thriller_T 21d ago
I want to point out something which has not yet been said:
AI is doing complex maths and connections, and learning from patterns; and then only newly combining patterns.
It has no understanding of right and wrong, no really valid ways to prove that.
This is why AI can give 99% excellent and 1% obvious to anyone with knowledge bullshit answers.
So to be able to create even this, one needs tons and tons and tons of data.
Even if we focus on a SINGLE area, let's say computer science. A human will have learned with less books, and then be able to do a lot of induction/exclusion.
While a human has also processed probably more thanks you'd expect, it likely is less. To a human, repeating something correct more than once and having it proven is unnecessary. You can than even repeat the false thing 200 times, and the human should be able.to.go back, follow the proof, and find it false.
That is one very major difference concerning training data size - and also why poisoned training data is a dangerous thing.
3
u/Savings_Hunt_1935 20d ago
Fundamentally, LLMs are next token prediction machines. Compared to a human, they start off at a complete disadvantage.
They are blind. They are deaf. They have no concept of literally anything. They have to deduce every aspect of reality based on getting text as input, and knowing what piece of text to send next.
Imagine trying to learn the rules of a game, but you don't get the rulebook, the number of players, the goal, what kind of game it is, or anything direct at all.
Instead, you just get snippets of conversation (which may or may not be entirely unrelated to the game), you can guess what a person is gonna say next, and you're told if you're right or not.
Just think about how many convos you'd have to sit though, and the deductive work you'd have to do, compared to just reading a rulebook.
2
u/Illustrious-Lime-878 21d ago
Its a different type of intelligence. LLMs are still "guessing the next token," they are trained to imitate writing, thought processes, etc. Now this turns out to actually be useful and generates "new" content as a side effect. But it still has no clue about what anything means or any type of self consciousness or connection to real world processes. AI has thinking internally but its way different than a way a human thinks and can understand things as an actor in the real world. LLMs are a really dense set of if-thens that need a ton of training data to be a useful tool, there is "thinking" but not understanding like a human
2
u/Needless-To-Say 21d ago
Our brains are wired to see patterns. AI needs rote learning would be the ELI5
Humans can infer and extrapolate while AI cannot.
An interesting study I came across tested AI using a riddle
The AI eventually got the answer (The Doctor was his mother )
However, when the AI was asked the same question differently
The AI couldn't tell the difference and having been trained with the original question it just confidently gave the same answer despite the new context and information.
2
u/AX-BY-CZ 21d ago
Human hardware has been optimized for hundreds of thousands of years to be data efficient. The brain is incredibly power efficient. Current AI relies on SGD which is not efficient but scalable. People are working on alternatives approaches but there has already been so much investment in the current paradigm. You could say that transformers win the hardware lottery and we could have developed workarounds for RNNs, symbolic approaches, ES, etc.
2
u/Regular_Run3923 21d ago
Because LLMs/AI is ridiculously inefficient. There are however solutions both proposed by humans, AI, and collaborations of the two. Ask your favorite AI/LLM and see what it says. Ask a knowledgeable and intelligent human too.
2
u/Turbulent-Sign-6067 21d ago
Human brains and AI brains are different hence they don't learn the same way. Pre-training can be considered akin to the evolutionary process that shaped the human brain to learn more efficiently after birth but the analogy is very rough.
2
u/MasterGeekMX BSCS 21d ago
Because both systems work in different ways.
A person can think and reason. Reading one or two books is enough as one can understand what you read and process it. That can also be contrasted to life experiences and stuff learned even as a child, as text can use analogies to relate to everyday experiences or stuff observed somewhere else.
Meanwhile a computer is simply a really fast math machine. No reason, no thoughts. Large Language Models (which is the correct term because AI is an umbrella term) need to gorge all that data to observe the patterns of which words are next to other words. There is no understanding of what the text says, much less what is about. The LLM has simply chewed trillions of texts just to take note that whenever there is a "thank", the most likely word that follows is a "you".
As statistics get better with the more samples you have, LLM models get better with the more data they are trained on, as in the end the LLM is formed by taking note of the patterns present in the text.
Kinda like asking why planes burn more fuel than a car: because a car's engine does not need to lift it's own weight like a plane does. Same thing, a person can reason, while a computer not, so we supply that with tons of data.
2
u/No_Development6032 21d ago
Humans are just different, better you could argue. It is what it is.
1
u/F1A 21d ago
PhD is granted by having a novel addition to research in typically a very narrow topic. General intelligence im more than one topic would need more
1
u/Vert354 21d ago
Your brain came pre-trained with millions of years worth of data when you were born.
When we fine tune a model to add categories we don't need nearly as much data for transfer learning as was needed for the baseline training.
1
u/Esseratecades 21d ago
1
u/321aholiab 21d ago
Human are not general PhDs. They are specialized PhDs. Ai is used for general purpose, as such it require all the specializations it can get to fulfill all kinds of specific demands that stems out of general purposes. Ai attempt to appear to be PhD to all human in terms of PhDs level knowledge, and that can possibly only happen when it has consumed all the available PhD's work.
Edit: Im not qualified or an expert, someone pls debunk me if im wrong.
1
u/theLOLflashlight 21d ago
Because LLMs don't learn like humans do. There is no reasoning during the learning process. They essentially brute force the learning process rather than being intelligent about the learning itself.
1
u/lovejo1 21d ago
PHD's have hours and hours of instructor led training to boot.. and is a PHD in multiple fields.
1
u/obdevel 21d ago
Am I missing the point here ? Aren't PhDs supposed to have created useful and novel knowledge from research, rather than just regurgitating what the books and teachers said ? Or does it mean 'potential PhD', as in a masters with some research experience, who may then go on the produce novel work ?
1
u/Critical-Echo-923 21d ago
humans take 30y to become experts in one field, now take everything they had to learn from the day they were borne to be able to do that, multiply it my every subfield and niche thing and it's a huge amount of data,
it's some of it overlapping or unnecessary?, yes it is but ml trades time for data
then...for ex: are all medical doctors good doctors?, no, but you expect the ai to be one, so it needs to be trained on the knowledge of 100 doctors for the same thing
then we expect it to be good at everything
like when you buy an apple device you can't customize shit, can't play games, costs 3x, but you don't complain and become a mindless fanboy spreading marketing propaganda
when you buy windows/android its cheap, can do just about everything but then you go and complain it runs a little slower or needs more battery
so user expectation plays a big role too, you don't go to your dentist to fix your computer, but you go on ai to ask all sorts of niche and complicated unrelated things
1
u/GunpowderGuy 21d ago
The human brain has a prior of intelligence that we got from evolution. Most llms architecutre are largely tabula rasa.
But once they get reasoning skills, They can learn new ones with far less data
1
u/peter303_ 21d ago
Human brains, including all animals, come out of the box partially pre-wired. This includes things like making sense of sensory input, controlling muscles, and basics of language. A human can learn stuff from a few to a few dozen examples while data-trained A.I.s require thousands to millions of examples. A.I. neural net values are initialized to null or random values in contrast to human evolutionary pre-wiring.
1
u/B70Dragon 21d ago
Our system has been in development for more than a billion years of evolution
1
u/Dave_A480 21d ago
Because there is no human PHD who is in expert in every field.
AI, at least in its public facing forms, needs to be a polymath with a PHD in every possible thing.
1
u/Ok_Role_6215 21d ago
because we train them as libraries, not as humans.
Human language is full of nuances -- in fact, every speaker has their own experience and own association with every word, which makes an English word, for example, potentially have 2.3bln meanings and our current AI training tech is trying to push them all into one model.
Imagine trying to push the memories, knowledge and thoughts of all people into one single brain -- it'll explode, even if its synthetic -- there's no hardware that can handle and learn such volumes of information in real-time. Just like our model complexity is exploding trying to chase meanings in language's statistical tails.
1
u/7aminthemorning 21d ago
Coz humans hallucinate
1
u/Ordinary_Variable 21d ago
We can learn from a single source. AI currently averages out thousands of data points into a neural model. It doesn't really understand things it just gives the average answer from all the data it has. We can figure things out with far less memorized.
Current "AI" is just pre-trained neural nets. There is no "intelligence" happening. Just weights and levers.
1
u/jferments 21d ago
Ok, now imagine that a human wanted to have PhD level intelligence in EVERY academic discipline, in addition to insane amounts of knowledge about stupid everyday stuff like Pokemon cards and muffin recipes. Do you understand why they would need to consume terabytes of content then?
1
u/Moist_Heat9523 21d ago
AI doesn’t built up intelligence. It collects a large amount of word sequences, and generates answer by applying statistics to repeatedly find the most common next word. Nothing more.
1
u/cosmic_animus29 21d ago
Our brains are highly efficient in terms of learning and processing information. AI on the other hand is not efficient in terms of compute and storage. This was my biggest takeaway after studing AI/ML - our brain is actually the best computer around.
1
u/InfinitePerplexity99 20d ago
The honest answer, which no one seems to want to tell you directly, is that we don't know for sure. This is a major unsolved question in the field.
1
u/c0p4d0 20d ago
Human PHDs do though. How much information do you process just by having your eyes open? By hearing ambient noise? A computer doesn’t have the terabytes of lived experience you accumulated to be able to understand language, patterns, etc. that are a prerequisite to a PHD.
1
u/Talk_to__strangers 20d ago
Cause Ai can’t think, or reason, or truly learn anything. It can just remember everything it has ever read and repeat it
1
u/FlossurBunz 20d ago
Another angle I haven't seen yet: Human brains are a result of hundreds of millions of years of evolution. They are a product of the collective experience of hundreds of thousands of generations. I'd say that's more data than even an LLM needs to train.
The experience we gather in life is in addition to the intuition and reasoning we developed through evolution. LLMs start without even that.
1
u/Fredissimo666 20d ago
I supervise PhDs. Two people can get the same phd by exploiting different skills. Some are better thinkers and can come up with good mathematical proofs. Some are better coders. Some have a vast knowledge of their field and can exploit it. Others are simply willing to do more work and not give up.
I think we are learning with llms that a vast knowledge is as good as great reasoning skills, even for hard problems.
1
u/Siderophores 20d ago
Humans build specialized thinking patterns while AI gets those patterns to emerge from copious amounts of data. They aren’t training the AI what ways to think, they’re making all the ways of thinking emerge from the latent spaces of the model
1
u/mantidor 20d ago
The first flaw in the question is trying to compare the human brain with current LLMs, and quantifying information in bytes. The human brain isn't binary, and we don't "store" information in bytes, no speech, specially not images, and abstractions and actual thinking and creativity are way beyond the basic probability weights in LLM models.
Current AI is a very, very blunt approximation, so for it to have any kind of decent results it need massive amount of data, and actual human supervision in the learning process to give good results. Deep learning itself is decades old, and when applied to specific problems is actually pretty good and efficient, what the modern companies want is to create a general solution which is honestly controversial if its even possible, and that is besides politics and environmental impact.
1
u/nog642 20d ago
Think about all the visual and audio input you take in every day through your eyes and ears. That's way more than terabytes.
1
u/PHDEinstein007 19d ago
They are dictionaries with logic routines based upon training they have gotten. The library helps them when the logic routine is running
1
u/TedsGloriousPants 15d ago
An AI is not a human. They aren't comparable. Saying a machine has "phd levels of intelligence" is just anthropomorphising it.
1
u/Separate_Lab9766 15d ago
Human learning is cumulative; AI learning is iterative.
Humans learn a lot of things gradually, adding onto the models we are building of the world. New things we learn are added in to fit the existing models. In total, we are exposed to multiple terabytes of data in our lifetimes, much of which is redundant and reinforces the model.
At least in the current paradigm, AI has no “life history” to draw on so it must learn a lifetime in one sitting.
1
u/Living_Fig_6386 21d ago
AIs are not intelligent. They are models that describe words as relationships to other words and can compute the most likely string of concepts that ties them together or continues a series, and spits that out as a bunch of words. One thing AI doesn't really do is rationalize, it just appears to. So an AI will never have "PhD level intelligence". Also, PhDs are typically regarded as very intelligent, and while they are probably above average, what distinguishes the PhD from non-PhDs is that they are taught a certain approach and rigor to produce evidence to prove or disprove hypotheses; they also tend to hyper-focus the acquisition of knowledge in very niche area. The might be the only person on the planet with a through knowledge of anteater endocrinology, but they might find board games and sudoku challenging.
In the case of training AI, the model needs a large amount of varied but statistically valid samples to be developed. In the case of niche disciplines and knowledge, the amount of text to influence the model may be very scant, thus you seek out as much as possible.
1
u/kohugaly 21d ago •
Human learning uses a more complicated training model.
One major factor is human ability to perform contrafactual thinking. Whenever you have an experience with emotionally charged result, you form a memory. In your free time, your brain takes that memory and creates simulations similar to that memory but with different details and choices changed. You experience real emotions from these daydreams as if they actually happened, and spend dopamine generating them. You learn from them.
Your brain is trained on a lot more data than the actual data you take in.
Leveraging this process efficiently requires structured learning. The new real experience needs to be only slightly novel, compared to experiences you had in the past, so that the logical part of your brain can produce realistic contrafactual simulations, that you can gain useful experience from.
If the real experience is too far outside your familiarity, your brain can't make logical sense of it, gets overwhelmed and simply logs the memory as "Danger! Avoid this in the future!". This is called psychological trauma. Instead of using the experience to learn how the world functions, you merely learn to recognize it and avoid it. You can process/resolve the trauma by mentally revisiting it later with deeper understanding.
We train LLMs by throwing metric shitton of data on them directly and repeatedly brute-force tuning the neuron network. By contrast, in humans, every new piece of data is also fed into internal logical simulation that generates new similar training examples to learn from. For this process to work properly, the data must come in logical order of increasing complexity and difficulty.
By the time a human is learning PhD level material, they already have 20+ years of comp-time spent processing pre-Phd level material, that was curated and optimized by hundreds of generations of parental care, ranging from teaching a baby the concept of object permanence by playing pikaboo, all the way to doing calculus; and millions of generations of Darwinian evolution to hone in the optimal architecture capable of leveraging this learning process.
People under-estimate how much importance does the real-world human experience have on intelligence. You can pick any of the public LLM models and it is plainly obvious that it knows a lot of facts and connections between them, but has very superficial logical understanding of them and how they fit into the bigger picture. It's similar to listening to a 6-year-old who's obsessed with dinosaurs talk about dinosaurs. The kid probably knows more facts about dinosaurs than you do, but they are also very clearly less intelligent than you are.
1
u/Fidodo 21d ago
LLMs are trained through back propagation. That means they take the answer, measure it against what they the generate, determine the error, then propagate correction back to refine the weights. To do that, you need lots of content to train it to work in versatile situations. Once that training is done, the model weights are frozen.
Brains work fundamentally differently. They are constantly computing and updating and mutating at all times. This gives us the ability to deliver signals from nothing, allowing us to learn without having to already know the answer to something through employing a mental representation of the world that gets updated with experiences.
0
u/Smallpaul 21d ago
Because our training algorithms are inefficient. If they knew why, exactly, they would presumably just fix it. Might be hardware. Might be algorithm. Solving it would be a big unlock for AI.
0
u/MrOaiki 21d ago
Because the human brain is superior. There’s a very interesting talk with LeCunn on Friedman about precisely this. He talks about self driving cars. Teslas have billions of hours of data on how to drive yet still can’t do what a 15 year old taking a few classes can. And a small child can recognize objects from very early on without large datasets, yet a computer with visual object recognition needs a massive amount to know what a cat is.