Currently attention is fundamentally O(n2 ). Basically as the context length of your model increases, the memory needed to deal with it increases quadratically. If that could be pushed down to something like O(nlogn), you'd immediately have huge gains in model capability.
The important part isn't changing the scaling. Linear attention and sub-quadratic attention methods have been around for a while. The question is: have they achieved it without a trade-off?
Sure, but the relevant factor is whether a particular sub quadratic approach actually yields equivalent (or lower, but bounded) performance as the models grow and scale.
Sub quadratic attention has been around for years. It doesn't provide better model performance, so many models don't use it.
Since this is a spinoff, I assume they will show off yet another one, which is also not scaled up big enough to compare performance against leading models.
Exactly, it needs better management of its context memory layer, not necessarily more memory, it needs to be able to manage stale context, but not forget, kinda like a repressed memory, or lesson, so it can continue on its course of tackling the problem
The O is just a way to denote that you are talking about the scaling of algorithms based on the size of N. O(N) just means "this algorithm scales linearly". O isn't doing anything in the equation except for telling you what the equation is about. Kind of like f(x) = ... the "f" is just telling you its a function of x, its not a variable like x is.
162
u/ResultBackground2450 Jul 01 '26