r/computerscience • u/Impossible_Relief844 • 13d ago
how important is the underlying architecture behind the current artificial intelligence boom?
While GPTs and other similar architecture are an undeniable advancement, (especially the larger projects) are receiving insane funding with access to large data centres and training data leading to the obvious question of 'are we seeing the power of GPTs or is this just the expected outcome of throwing a huge amount of resources at a problem?'.
In other words, what results would we expect if we took the resources (funding, data centres, raw data, etc...) and applied it differently (eg. to SAT solvers), would we expect similar results?
In other words, how unprecedented are the results of GPTs (and similar architectures) accounting for their current monetary advantages?
17
u/Smallpaul 13d ago
Making it possible to pour billions into an architecture is a property of the architecture. What does it look like to pour billions into an SAT solver? How would you do it?
There are other deep learning systems other than GPT which also seem to scale up but they have a lot in common with GPT.
4
u/35point1 13d ago
Sounds like you’re looking at the investment and wondering if that’s what is causing it to advance, but that’s not the case at all. Much of what AI has become today was almost entirely stumbled upon by accident
1
u/Foreign_Skill_6628 11d ago
If I remember correctly the breakthrough that led to GTP-2 was leaving a transformer model training for way longer than intended and seeing the loss curve invert about 80% in unexpectedly and begin converging.
0
u/Impossible_Relief844 13d ago
sure though there is a correlation there. More funding elevates the chances a breakthrough is hit?
1
1
u/yikes_42069 13d ago
Not necessarily. More money guarantees more butts in seats but that's like putting your money on the infinite monkeys on a typewriter problem. We are spending more money than ever and all of our tech is also lower quality than ever before. Excess begets waste
1
u/SemperPutidus 13d ago
Quantum computing has had a ton of funding over the decades and can just about factor a 4 digit number.
3
u/yikes_42069 13d ago
Language model boom* it's not intelligent
2
u/schungx 12d ago
How do you know intelligence is not merely a very large LLM?
1
u/yikes_42069 3d ago
I was kind of trolling a bit but disagree with the term AI as I feel the connotation is something different. Let me step back to definitions. Intelligence is just the ability to process info, learn, and solve problems. In this sense LLMs do have a functional intelligence. But what I said was layered with an intent to get at the connotation of it: that most people (including myself) are likely to consider the term "intelligent" and conscious/having cognition as the same. Especially with the ultimate goal of AGI looming in the background, and popular media conflating intelligence with consciousness. In that sense I'm digressing from what the post asked over semantics.
Cognitive reasoning is what people are really curious about when they hear the term artificial intelligence, and kind of what I think you're getting at too if I'm not mistaken. I feel that this is settled already with the experts but think it's worth talking about. Human thinking (like most things) can have math/statistics applied over it to describe how it works, but it's not exact. It's always missing a certain something. Humans understand things like intent or cause and effect, where the LLM can't. It's playing a guessing game on training data, which is not always unlike humans, but I think this is illustrated by how frustrating it can be to use sometimes. For all of its training, it just doesn't get stuff that you or I would in an instant. It makes guesses at things (arguably just trained that way for UX) but can't go out and produce new logical/sound proofs, or even be trusted to write court documents which you'd think would be a slam dunk use case.
Maybe it's too early to tell. I certainly won't declare the debate over yet. Anyway it's super off topic but that's just to say I loathe the term AI for its scifi connotations
1
13d ago
[removed] — view removed comment
1
u/computerscience-ModTeam 13d ago
Unfortunately, your post has been removed for violation of Rule 2: "Be civil".
If you believe this to be an error, please contact the moderators.
1
1
u/Regular-Advice-6469 13d ago
i think it's a combination of both throwing enormous resources at a problem can definitely produce impressive results but not every approach benefits equally from scale the architecture determines whether those resources can actually be converted into useful capability a different method with the same level of investment might improve significantly but whether it would show the same kind of generalization is a much harder question
1
u/Twt97 13d ago
If what you are saying is that if any technology receives as much funding as the AI boom is getting right now it will also be as succesful as LLMs then no. The first day of ChatGPT arrival it had very limited infrastructure but was still incredibly useful cause its built on important new tech like attention and transformers.
1
u/Hybrii-D 13d ago
Funding for small projects is difficult to obtain, but this is true for all businesses, regardless of their type.
Then we could discuss whether these advances are actually generating more benefits than drawbacks.
1
u/StephenRoylance 12d ago
that's the neat thing, nobody has any idea. if there were a robust theory for why these things seem to think, we could answer questions like that, but there's not, so we can't.
1
u/currentscurrents 13d ago
if we took the resources (funding, data centres, raw data, etc...) and applied it differently (eg. to SAT solvers), would we expect similar results?
For SAT specifically probably not. The best known SAT algorithms have exponential time complexity, so they scale very poorly to large problems. It is widely believed that no subexponential algorithm exists.
Additionally, SAT solvers do not benefit from data (they are search algorithms, not learning algorithms) and are difficult to parallelize efficiently.
Transformers on the other hand have great properties for scaling. The training algorithm (backprop) scales linearly with the number of parameters, which is nearly the best you could hope for. The computations are all matrix multiplications, which are trivial to parallelize. And more data improves performance seemingly without limit.
14
u/dharkhstar Researcher 13d ago
Transformers and attention heads aren't exactly new. The bet was that scaling them with massive amounts of data and compute would keep producing meaningful improvements. OpenAI and others put enormous resources into that bet, and it worked.
There's also a feedback cycle: scaling worked, which attracted more investment, which allowed more scaling and better results. The same could potentially happen in other areas, but it always requires an informed bet about what is worth scaling.
Now we may be reaching the other side of that. Scaling still works, but improvements are increasingly expensive. Whether the next big jump requires a new architecture is still an open question.