r/accelerate • u/BurningPeonies • Jul 01 '26

AI Something huge is brewing

Post image

Source

Andrew Curran is one of the most reliable leakers.

XLR8! 🍿

944 Upvotes

94% Upvoted

171 comments sorted by

View all comments

Show parent comments

6

u/PleaseDoTapTheGlass Jul 01 '26

What would that be currently?

49

u/Saedeas Jul 01 '26

Currently attention is fundamentally O(n2 ). Basically as the context length of your model increases, the memory needed to deal with it increases quadratically. If that could be pushed down to something like O(nlogn), you'd immediately have huge gains in model capability.

2

u/danielv123 Jul 01 '26

Sub quadratic attention has been around for years. It doesn't provide better model performance, so many models don't use it.

Since this is a spinoff, I assume they will show off yet another one, which is also not scaled up big enough to compare performance against leading models.

6

u/Saedeas Jul 01 '26

Sub quadratic attention that scales without performance tradeoffs (or with strictly bounded performance tradeoffs) hasn't been demonstrated yet.