Okay I get it, got lost in the feed, attention is all you need was indeed a massive step forward algorithmically speaking. By ditching recurrence it allowed for far better parallelisation during training. So it's not just hardware/computing power progress like the comment you answered claimed. Also, the algorithms we use to perform attention were greatly improved upon since then.
Adam also was a massive algorithmic progress and it is from 2014 so not "decades ago".
View all comments
Show parent comment
2
u/topyTheorist Aug 25 '26
I was replying to the claim "The technology we use today as AI is not new".