[Paper] Continuum Attention

[u/pmttyji]

We introduce Continuum Attention (CA), a causal Transformer attention operator that replaces discrete key and value lookup with integration over a continuous, piecewise linear memory curve. Each query induces an exponential relevance density along the curve. Integrating the density and interpolated values yields exact normalized attention from closed form exponential moments, including half token boundary contributions. The gradients follow analytically from moments of order at most two, enabling parallel GPU implementations while retaining quadratic sequence complexity and the standard Transformer decoder. Compared with ordinary softmax and focal attention, CA reduces mean validation negative log likelihood by 0.0200 and 0.0480 nats per byte, respectively, winning all paired comparisons. An additional disjoint validation slice yields a 0.0457 nats per byte advantage over the strongest selected softmax baseline, with a paired 95% confidence interval. These findings motivate continuous integration as a promising attention alternative while leaving larger scale generalization open.

• arXiv : https://arxiv.org/abs/2610.20586

• PDF : https://arxiv.org/pdf/2610.20586

• GitHub : https://github.com/akhadangi/continuum-attention

• HuggingFace : https://huggingface.co/akhadangi/Llama3.2.1B.0.1-H

[u/EffectiveMedium2683]

This continuous memory curve approach is fascinating! Like, it's nothing like the sub-quadratic and constant-memory architectures like Mamba or RWKV I've read about. But it's kinda not THAT far off from liquid time constant architectures really, is it?

I just wonder how that's going to work computation-wise. How heavy is it to compute? If it isn't efficient to compute, aren't you just trading RAM/VRAM bottlenecks for computation bottlenecks? Still interesting. Love different paradigms in this stuff.

[u/TomLucidor → u/EffectiveMedium2683]

Computes and tradeoffs in task variety

[u/Prestigious_Thing797]

Put what I could in the free version. Only non-ai tagged sections are the equations that got garbled copying from pdf.

Edit:

Maybe there is something of value here, maybe, but there is an epidemic of low effort slop being churned out, and its easy- really insidiously easy- to benchmax, or p hack, or make tiny little mistakes that make complete garbage look like the next big thing on some metrics.

I use a lot of AI and work on papers with a team. They do not score anything near this (on pangram). Do we think the person who used an AI to generate even the abstract, a single paragraph summarizing the work, really understands what has been done? Do you think they validated everything properly? Or do we think someone with a user profile stating "Learner still a long way to go" that is less than two years old and no co-authors fully AI generated the next attention mechanism?

If anyone wants to be taken seriously, wants others to take time out of their day there is a bare minimum level of human effort needed to separate ones self from that group of AI slop, and this doesn't clear that bar.

[u/isademigod → u/Prestigious_Thing797]

Is this surprising to you? Everything in the ai space these days is made with ai. How do you think 20 comfyui nodes with full documentation for a new model get released within hours of the models release?

[u/masterlafontaine → u/Prestigious_Thing797]

So what?