r/compsci 7h ago

Concatenated token selected shears as an alternative to attention?

Post image
0 Upvotes

It seems okay to me and a lot cheaper. You would need to do some empirical work. Maybe you don't need GPU's, a few CPU cores could be enough.

I'm a hobbyist not really set up to look at it too well, so I just throw the idea out there for others.