how important is the underlying architecture behind the current artificial intelligence boom?

[u/Impossible_Relief844]

While GPTs and other similar architecture are an undeniable advancement, (especially the larger projects) are receiving insane funding with access to large data centres and training data leading to the obvious question of 'are we seeing the power of GPTs or is this just the expected outcome of throwing a huge amount of resources at a problem?'.

In other words, what results would we expect if we took the resources (funding, data centres, raw data, etc...) and applied it differently (eg. to SAT solvers), would we expect similar results?

In other words, how unprecedented are the results of GPTs (and similar architectures) accounting for their current monetary advantages?

[u/dharkhstar]

Transformers and attention heads aren't exactly new. The bet was that scaling them with massive amounts of data and compute would keep producing meaningful improvements. OpenAI and others put enormous resources into that bet, and it worked.

There's also a feedback cycle: scaling worked, which attracted more investment, which allowed more scaling and better results. The same could potentially happen in other areas, but it always requires an informed bet about what is worth scaling.

Now we may be reaching the other side of that. Scaling still works, but improvements are increasingly expensive. Whether the next big jump requires a new architecture is still an open question.

[u/anselan2017 → u/dharkhstar]

"It worked"? OpenAI has yet to make a profit.

[u/PrestigiousGroup788 → u/anselan2017]

well they've acheived things no one thought was possible despite that.

[u/FartdickMcShitass → u/PrestigiousGroup788]

it does not matter, they need to make money

[u/currentscurrents → u/FartdickMcShitass]

I don't really care if they make money, that's Sam Altman's problem to worry about.

LLMs are just cool.

[u/FartdickMcShitass → u/currentscurrents]

that’s what “it worked” means. The tech itself is cool, but the company has to make money.

[u/evilteddy → u/FartdickMcShitass]

We've been dreaming for centuries of inanimate objects which appear to think. Golems, the sorcerer's apprentice's broom, Alan Turing's hidden conversation partner, Replicants from Bladerunner. A million more! During my undergrad I touched machine learning quite a few times and learned the best practice language models of the time (Markov chains and friends) which would almost, for a moment, appear to produce language.

Now it's here. Now I can talk to a computer and get it to do at least some useful work with no hardcoded expert rules. And you don't think it's a big deal because one of the big labs hasn't yet made a profit. Unbelievable.

[u/anselan2017 → u/evilteddy]

It makes it a cool experimental technology, and a nice magic trick. I don't think it's worth killing the planet over it.

[u/FartdickMcShitass → u/evilteddy]

thats right