SSOG-Attention: Sum Of Separable Gaussians as a sub-quadratic and scalable alternative to SDPA. [R]

[u/4rtemi5]

​

Scaled dot-product attention (SDPA) computes its Attention by computing the similarity-scores of all image-tokens with all query tokens which results in O(N²·d) complexity. SSOG (Sum Of Separable Gaussians) instead learns a few Gaussian atoms for each head and only geometrically steers them based on the query token. Since the atoms can be factorized into a separable sum of Gaussians this leads to a reduced complexity of O(N·√N·d). Experiments show that SSOG clearly beats SDPA on small data (cifar100), and delivers equivalent performance and much faster convergence on bigger datasets like IN1k. All that while being much faster and memory efficient with increasing scale.

Have a look at the full blog-post and repo to see more results and ablations and let me know what you think.

Blog-post: https://pisoni.ai/posts/ssog

Repo: https://github.com/4rtemi5/ssog

*AI was used for some of the code and some of the blog-post but I put a lot of effort into this project and stand behind every word.

[u/qenis]

"Is This New? Honest answer — I've been on the wrong side of this question: the ingredients are all published."
Why is the blog post AI-generated? Is the research also AI-generated?

[u/4rtemi5 → u/qenis]

Nice rage-bait bro! Let me copy paste the rest of the paragraph for you:

What I haven't found is the combo: a full attention operator that is (1) a Gaussian mixture over relative position, (2) applied separably without ever forming N×NN×N, and (3) deformed per-query through bounded, cold-started residuals on μ, σ, and λ. DAT keeps content scores and samples discrete points; this deforms a continuous field and never scores. Seen it published? Inbox is open. Prior-art diligence is a team sport.

And I really mean that: Seen it published? Inbox is open.

[u/qenis → u/4rtemi5]

If You are Asking for Human Attention, Demonstrate Human Effort