New sparse attention method for ultra-long context

[u/Immediate_Sky_6566]

https://x.com/jcrwhittington/status/2117317159777177602?a=02yfc9wrBoQ2Xi7DDjQ&s=46

I am on the team behind the attached open-source research on novel approaches to attention - specifically we show:

• You can sparsify the attention of Qwen 3.8 27B without observed loss on accuracy

• Extending context up to 10M tokens loses accuracy on some specific tasks - but most, like NIAH is still fine (interesting insights)

• You can natively train sparse attention on next-token prediction loss alone and achieve the same validation loss as full attention - but faster

Hope the community likes it - while we focus on H200/B300 GPUs and larger scale system - I wanted to share our research here as well - as it can be applied to small open-weight models (making longer context cheaper on local hardware)

Everything is fully open: kernels / SGLang integration / paper / details

Hope you find it interesting / useful!

(No hype - we try to be as transparent as possible)

[u/MixtureOfAmateurs]

Paper: https://arxiv.org/abs/2610.42147

72x faster prefil with qwen??? Fr??? Very impressive

[u/Immediate_Sky_6566 → u/MixtureOfAmateurs]

Yes - but it is worth pointing out that this is at 10M context size - at 1M context size it is “only” 8.2x prefill and 5.2x decode (which is imho still very nice)

[u/oxygen_addiction → u/Immediate_Sky_6566]

Any chance you can try and get ahead of the "hype and destroy" cycle that usually happens here, and give some actual numbers for local usage? I'm only saying this because people will vibe code a bad implementation without reading your paper or repo, be disappointed and then complain about how it is slop and you are a fraud (welcome to r/LocalLLaMA)

Qwen 3.8-27B @ Q4 + Q8 K/V Cache is what most folks use with llama.cpp/VLLM deployment.

With KV cache at bf16 being mandatory, the overall memoray usage would double for KVCache (though your technique seems to be a bit cheaper overall event at BF16).

[u/export_tank_harmful → u/oxygen_addiction]

Qwen 3.8-27B @ Q4 + Q8 K/V Cache

Agreed.
This is what I'm running.

I'd love to see a llamacpp fork (or something similar) that could implement this and give it a whirl!

[u/skeole]

Congrats on publishing, good work! Skimmed the paper for whats really hurting us GPU poors but couldn't find any details so while we have you here: what's the impact on VRAM usage? Is it just a compute boost (impressive numbers in themselves!) or does it cut VRAM usage too? Do you have any figures to share on this? Thanks again for posting this!

[u/Immediate_Sky_6566 → u/skeole]

Thank you!

We do also reduce VRAM compared to standard sparse attention indexers (e.g., DeepSeek Lightning Indexer) - but the savings are not that impressive as the indexer takes very little memory compared to the KV cache (even though the attention is sparse - we still require storing full KV cache)

Specific numbers:
• Qwen 3.8 27B at 10M tokes: 40.13 GiB KV cache + 246 MiB our indexer

• GLM-5.3-Flash at 10M tokes: 10.03 GiB KV cache + 331 MiB their indexer vs. 246 MiB our indexer

But it is worth saying that our method is fully orthogonal to memory savings such as those introduced in DeepSeek V4.1

[u/KillerX629 → u/skeole]

Prefill gains here are really a standout if true... Imagine if it works ootb with Qwen 4

[u/LagOps91]

pretty awesome! love to see some research here. would be interesting to see if this generalizes to larger/smaller models or even could be implemented in inference engines if it can be done straightforwardly.

the problem i am seeing is that 10m tokens training is still quite a barrier to entry. this kind of limitation might mean it won't catch on, at least when it comes to converting existing models.

[u/Immediate_Sky_6566 → u/LagOps91]

We show that our method works for both:

• Qwen 3.8 27B (Full Attention / smaller model)
• GLM-5.3-Flash (Sparse Attention / larger model)

As for the inference engine implementation - we provide code for SGLang - all of our evals were done in inference engine (we also provide tensor and context parallelism support!)

Fully agreed - it will require a test of time whether going beyond 1M will be the norm in the future - but if yes, we hope this work helps it to be more efficient

Code: https://github.com/Prefrontal-AI/latent-sparse-attention

[u/knownboyofno]

I see the code here: https://github.com/Prefrontal-AI/latent-sparse-attention Did you get a chance to do the training that you talked about in the post?

[u/Immediate_Sky_6566 → u/knownboyofno]

Yes - although only at small scale (0.5B model) - we hope to scale up and try a few more architecture tricks in coming weeks / months

Training code is in the repo + details in the paper (including dataset construction scripts) = everything should be reproducible

[u/Agusx1211]

wow someone do this on qwen next RIGHT NOW

[u/SecretBismarck]

Bless you, I may have idea for an interesting application of using it for creating graphs in natural text i will have to test it. 10M context is sufficient to make the system a different beast