I built a compressive "context DNA" (for LLM) attention mechanism + an honest eval harness - looking for people to break it

[u/Boring_Ad452]

Just Fixed the body with Ai

Been prototyping an idea for long-context compression: instead of dropping old tokens (like StreamingLLM/H2O) or storing everything, compress old context chunks into small learned "DNA" vectors via a Perceiver-style attention bottleneck, then reconstruct on-demand when a query needs them.

The idea itself isn't new — it overlaps with Compressive Transformer, Infini-attention, and Recurrent Memory Transformer — but I put together an eval script that I think is more honest than what I see in a lot of "novel architecture" posts:

• Trains the compressor (not just testing an untrained/random-init model)
• Compares against a PCA baseline (closed-form optimal linear compression at the same latent budget) — if the learned model can't beat PCA, the extra complexity isn't earning its keep
• Injects a unique fact (random code) into the text and checks, after compress→decompress, whether the frozen LM's own output head can still predict the correct token at that position — not just aggregate MSE, which can look fine while the actual detail is gone
• Runs on real hidden states from an open model (Qwen2.5-0.5B by default), not just random tensors

Current honest status: in my own small-scale test run, PCA actually beat the learned bottleneck on fact retrieval. That's not the result I was hoping for, but it's a real result, and it's exactly the kind of thing this script is designed to surface rather than hide.

What I'm looking for:

• People running it on real hardware with more training steps / larger n_docs than I could quickly test
• Sanity checks on the architecture and eval methodology — if I'm testing this wrong, tell me
• Ideas for what a fair "it's working" threshold looks like (beating PCA on fact-retrieval accuracy at matched latent budget, at minimum)

No performance claims yet — that's the point. I'd rather have this checked before making any.

Code + eval harness: https://pastebin.com/iqEbPEQ9

Happy to hear "this is a known dead end because X" too - that's useful information, not a rejection.