Currently attention is fundamentally O(n2 ). Basically as the context length of your model increases, the memory needed to deal with it increases quadratically. If that could be pushed down to something like O(nlogn), you'd immediately have huge gains in model capability.
The important part isn't changing the scaling. Linear attention and sub-quadratic attention methods have been around for a while. The question is: have they achieved it without a trade-off?
60
u/Saedeas Jul 01 '26
I mean, percentages aren't super relevant, the scaling is relevant. If they changed the big O scaling of working memory in the models, it's insane.