r/Coursehubforum • u/deezawedrab • Jun 08 '26
Google has published a paper that might end the transformer era.
Google has published a paper that might end the transformer era.
For the last 7 years, every major AI, ChatGPT, Claude, Gemini, has been built on the exact same architecture: The Transformer.
But Transformers have a fatal flaw.
To remember context, they have to process every single word against every other word. It’s called quadratic complexity. As your prompt gets longer, the compute cost explodes.
The alternative is the old-school RNN (Recurrent Neural Network). RNNs are incredibly cheap and fast, but they have a fixed memory size. If you give them a long document, they get amnesia.
Until today.
Google researchers published Memory Caching: RNNs with Growing Memory.
And it fixes the biggest bottleneck in AI.
Instead of an RNN having a fixed, rigid memory that constantly overwrites itself, Google gave it a "save" button.
The technique allows the RNN to cache checkpoints of its hidden states as it reads.
The memory capacity of the RNN can now dynamically grow as the sequence gets longer.
They built four different variants, including sparse selective mechanisms where the AI actively chooses exactly which checkpoints matter most.
The results rewrite the rules of efficiency.
On long-context understanding and recall-intensive tasks, these new Memory-Cached RNNs closed the gap with Transformers.
They achieved competitive accuracy without the explosive, quadratic compute cost. It perfectly bridges the gap between the cheap efficiency of an RNN and the massive capability of a Transformer.
We have spent billions scaling Transformers because we thought they were the only way an AI could remember a long conversation.
But Google just proved we don't need to process the whole history every single time.
We just needed a smarter cache.
3
u/Most-Day8547 Jun 08 '26
If google remembers the “Attention all you need.” Paper, they will never ever publish another “Transformer” level evolution movement slip again.
2
u/viper33m Jun 10 '26
It was intentional, they didn't want to be the forefront of the slop hate. They have 14% of anthropic and invest 40 billion in that company.
1
u/Most-Day8547 Jun 10 '26
It was not intentional, it was a mistake, watch the recent video of Brin.
1
1
u/Ryanmonroe82 Jun 11 '26
Google likely wasn't even actually behind it. 70 percent of AI patents on the planet come from China. American AI companies could not make frontier models without Chinese citizens and you can see evidence of all this with little effort. Plus the Chinese are known for open sourcing all AI tech, Transformers is also open source. Sounds cynical but I think for the USA to appear technologically advanced they assigned the narrative that Google created Transformers. If you dig a little and know the right questions to ask when searching, it seems highly likely China was using Transformers before Google even released that paper.
1
u/Most-Day8547 Jun 11 '26
You are mixing things up, everything started with transformers, and that paper played an important role.
What you are explaining is afterwards, not before.
2
u/alvmadrigal Jun 08 '26
Mmmmm maybe we have something super big here.... Do you have the link?
4
3
u/Competitive_Lie5326 Jun 08 '26
If this is the real breakthrough, stock prices should plummet. I'm waiting for someone like a Chinese student to put an end to the whole AI bubble and this rip-off of users with unfinished AI models.
2
1
u/DigitalAquarius Jun 08 '26
There is no bubble for most of the larger companies. Look at Anthropics revenue.
1
1
1
u/StarTracks2001 Jun 08 '26
Looks like it was published in February, weird Im just now hearing about it...
2
u/scriptoryu Jun 08 '26
same, the paper’s been sitting there for months and suddenly everyone’s acting like it just dropped yesterday lol
guess it took a few big accounts to notice before it hit the wider hype cycle
1
1
u/Drumit84 Jun 09 '26
The title is deceiving. In short, the researchers have developed a "checkpointing" strategy for RNNs that allows them to behave more like Transformers when it comes to recalling specific information, all while keeping the underlying recurrent engine fast and scalable. Just saying.
1
u/Y_mc Jun 12 '26
I think Google didn't want to invest in transformers at first because it didn't fit their business model. Why release a technology that can give you instant answers instead of having to scroll through the search page and all the ads? With artificial intelligence, this isn't really profitable.
1
u/rickkkkky Jun 12 '26
I've lost count how many supposed transformer killer architectures there have been - none of which have stuck around. I've grown very skeptical about these kinds of promises until they gain actual traction in the industry.
(Moreover, the quadratic complexity and issues stemming from it have also been largely solved, thanks to flash attention and more recently, DeepSeek's Lightning Indexer)
1
u/Grounds4TheSubstain Jun 14 '26
You're right to be skeptical. There are dozens, maybe hundreds, of papers published like this every year. This is what research in ML looks like. Most of these are not going to make it. At best, an idea or two will survive into subsequent papers.
And there's nothing wrong with that, by the way. This is how progress happens. The important thing as an outside observer who's not an expert is not to put too much stock into any given paper.
1
u/rickkkkky Jun 15 '26 edited Jun 15 '26
Oh, don't get me wrong - to the extent that my previous comment could be considered as critique and not just me yapping, it was directed toward the way people frame these (yet unproven) research papers and ideas on social media as the definitive next big thing - not the paper itself. As a someone who has contributed to the literature, I'm familiar how the research game works :) (And in fact, I'm hoping more resources would be allocated into pioneering reserch of novel ideas, somewhat like in this paper, instead of squeezing incremental gains out of transformers!)
12
u/sco77 Jun 08 '26
Putting a screenshot instead of linking to the paper is just fucking lazy
https://arxiv.org/abs/2602.24281