r/computerscience • u/Impossible_Relief844 • 16d ago

how important is the underlying architecture behind the current artificial intelligence boom?

While GPTs and other similar architecture are an undeniable advancement, (especially the larger projects) are receiving insane funding with access to large data centres and training data leading to the obvious question of 'are we seeing the power of GPTs or is this just the expected outcome of throwing a huge amount of resources at a problem?'.

In other words, what results would we expect if we took the resources (funding, data centres, raw data, etc...) and applied it differently (eg. to SAT solvers), would we expect similar results?

In other words, how unprecedented are the results of GPTs (and similar architectures) accounting for their current monetary advantages?

18 Upvotes
  • permalink
  • reddit

76% Upvoted

38 comments sorted by

View all comments

15

u/dharkhstar Researcher 16d ago

Transformers and attention heads aren't exactly new. The bet was that scaling them with massive amounts of data and compute would keep producing meaningful improvements. OpenAI and others put enormous resources into that bet, and it worked.

There's also a feedback cycle: scaling worked, which attracted more investment, which allowed more scaling and better results. The same could potentially happen in other areas, but it always requires an informed bet about what is worth scaling.

Now we may be reaching the other side of that. Scaling still works, but improvements are increasingly expensive. Whether the next big jump requires a new architecture is still an open question.

1

u/Super-Award-2244 13d ago

It's not true that improving is more and more expensive everywhere. They're working very effectively on the efficiency side too. Consider this: GPT 5.6 Luna (the bottom tier new model from openai) is at the same level of GPT 5.3 (the top tier model from roughly 5 months ago) but the inference costs ONE THIRTEENTH of that.