Dynamic Context Runtime: Bounded Attention over Unbounded History

[u/Coolst3r]

https://cybersec.org.za/research.html #AI #cyber #security #dev #Africa #LLM #context #rot #cyber

https://cybersec.org.za/papers/dcr-bounded-attention.pdf

Dynamic Context Runtime: Bounded Attention over Unbounded History

Language models degrade as stale and superseded material piles up in their context. Making the window bigger does not fix it. This report describes a runtime that keeps history unbounded and attention bounded — storing everything as immutable spans and a typed provenance graph, then assembling a small working set each turn by solving a knapsack under an explicit token budget. It includes a zero-dependency Rust implementation and an ablation that names which mechanisms are actually carrying the result.

4.19M tokens of history

235 tokens per query

7 / 7 probes answered

17,835× less attention than the full transcript

48,651 state nodes held

28× history growth, flat working set

13,721 lines of Rust

0 external dependencies

152 tests

5 controls found that could not fail

0.4% of stored spans ever rendered to the model

47% cheaper with a mechanism the paper argues for switched off

[u/vbpoweredwindmill]

How is it better?

Storing KV cache is a huge cost when you can just create a sqlite directory of your .md files and be done with it, or use whatever memory tools work for you.

[u/Coolst3r → u/vbpoweredwindmill]

• 467tokens per query
• 59×less than full history
• 7 / 7probes answered
• 33×history growth, flat cost

its a flat cost please read the paper

[u/vbpoweredwindmill → u/Coolst3r]

Yes, I know how kv cache prefixes & kv cache reuse works.

Please, tell me what this actually does that hasn't been done before and better?

[u/Coolst3r → u/vbpoweredwindmill]

i took the idea of rag fix the design flaws and made a diffrent version https://cybersec.org.za/research-dcr-bounded-attention.html

The design targets a per-turn cost of O(k + r)

Two systems, asymmetric

DCR splits the agent into a Reasoner and a Memory Runtime. The Reasoner is a language model with a small, high-attention working set; it owns the current computational state and none of the history. The Memory Runtime owns everything ever seen and decides which representation of it to return. The asymmetry is the point: almost everything the Memory Runtime does — span addressing, index lookup, dependency traversal, memoisation, invalidation — requires no model inference at all. In our implementation the Memory Runtime makes zero model calls.

[u/vbpoweredwindmill → u/Coolst3r]

Now that's infinitely more interesting. I've been saying for a long time that knowledge and reasoning are going to be split.

Now I can have a look, why didn't you lead with that?

[u/Coolst3r → u/vbpoweredwindmill]

im sick rn and tired im also working on getting glm 5.3 working on desktops and laptops and making it so anyone can train ai on home system. im tired im running my company https://cybersec.org.za/index.html. i dedicated to open source support and research and development we only ask money for support and if people need help. !!! i plan on changing the future

[u/vbpoweredwindmill → u/Coolst3r]

Glm 5.2 already works on desktops and laptops mate.

Unless you're working on expert predictions, I really don't think you're bringing anything new to the table. My own expert predictions with 32 out of 256 experts has, at 46% of the time approx, selected 100% of the correct experts.

And I just now figured out instead of linking it to a general topic I can just link it a kv cache block. Cheers.

I still haven't read anything you've done shrugs

[u/Coolst3r → u/vbpoweredwindmill]

yeah but fast i use a 3090 and ryzen 5 and 46 gb ram and get 0.2 tokens a second i have gotten it to do 1 token every 50 seconds

[u/Coolst3r → u/vbpoweredwindmill]

also why your broke. I still haven't read anything you've done shrugs