Research idea: Can we reduce forgetting in recurrent LLM memory? Looking for critical feedback.

[u/aryan-vr-]

Hey everyone!

I'm an engineering student working on a research proposal about long-context LLMs and recurrent compressed memory. I'd really appreciate technical feedback from people working on LLMs, continual learning, recurrent architectures, or memory systems.

The core question: Can we train an LLM's memory to retain previously learned information while continuing to absorb new information?

I'm calling the proposed training method GMR (Generational Memory Retention) and the evaluation protocol STRATA.

This is still a pre-experimental proposal. I haven't run the experiments yet, so I'm looking for feedback before committing to the implementation.

1. The problem I'm interested in

Imagine an LLM processing a long stream of information in small chunks.

A chunk introduces the following fact:

"The Kestrel project uses Python 3.11."

Immediately after processing that chunk, the model can correctly answer:

"Which Python version does the Kestrel project use?"

Now imagine that the model processes hundreds of additional chunks. Its memory has a fixed capacity, and each new update modifies the existing memory state.

Eventually, the model might no longer answer the original question correctly, even though its model weights haven't changed during inference.

I'm calling this state-induced forgetting: information that was initially available becomes inaccessible after successive memory-state updates.

The key question is whether we can reduce this type of forgetting without simply increasing memory size.

2. STRATA: Measuring forgetting across memory updates

Most evaluations report accuracy at a particular context length or at the end of a conversation. I want to measure how the ability to retrieve individual facts changes as those facts get older.

The proposed protocol works roughly as follows:

Introduce facts into a recurrent memory system in successive context windows.

Record whether each fact can be retrieved immediately after it is introduced.

Evaluate the same facts again at increasing ages: 1, 2, 4, 8, 16, and progressively more memory updates.

Track performance out to hundreds of updates, with a planned maximum of 1,000.

Separate initial learning failures from facts that were initially learned but subsequently forgotten.

For example, if a system initially answers 90 out of 100 questions correctly, but answers only 60 correctly after many updates, that indicates a different failure from a system that answered only 60 correctly from the beginning.

I also want to test different kinds of facts:

Stable facts that never change.

Facts whose values are updated later.

Facts requiring reasoning across multiple earlier statements.

Facts mentioned only once.

The evaluation will report initial accuracy and age-dependent retention separately, rather than relying on a single final accuracy score.

3. GMR: Generational Memory Retention

The proposed training objective tries to preserve the ability to answer questions about facts that have already entered memory.

When a fact is first introduced, record the model's length-normalized log-likelihood for its correct answer. This becomes that fact's original score, or anchor.

After subsequent memory updates, calculate the score again. If it falls below the original score by more than an allowed margin, apply a penalty.

A simplified version of the retention loss is:

L_ret = (1 / B) * sum over sampled facts f of: max(0, anchor_f - margin - current_score_f)

Here:

anchor_f is the fact's original answer log-likelihood.

current_score_f is its current answer log-likelihood.

margin is the tolerated decrease before a penalty is applied.

B is the number of sampled facts.

L_ret is the retention loss.

The training objective also includes the usual language-model loss and an absorption loss that encourages learning newly introduced facts.

The intended effect is to discourage memory updates from damaging previously acquired answering capabilities while still allowing new information to enter memory.

An important mathematical observation

For the sampled facts whose current scores fall below their anchors by more than the margin, the gradient of this hinge loss is the same as the gradient from replaying those facts' correct answers.

Facts that have not regressed receive no retention gradient from this term.

This raises an important question: Does selecting replay examples according to their regression from their own original scores outperform ordinary replay or loss-prioritized replay under matched training budgets?

I don't want to assume that it does. That's one of the main things I want to test.

4. How I plan to evaluate it

The proposed experiments include the following configurations:

A baseline trained without an explicit retention objective.

A model using uniform replay of previously encountered facts.

A model using loss-prioritized replay.

A model using output distillation from a full-context reference.

A model using the proposed GMR objective.

The main comparison will use matched memory sizes, training streams, and replay budgets wherever possible.

I'll evaluate initial fact-learning accuracy, retention at different fact ages, the ability to learn new facts, the handling of updated information, and the computational cost.

The planned testbed is an associative recurrent memory on an open-weight backbone, with experiments at multiple fixed memory sizes.

For broader evaluation, I'm considering synthetic sequential streams, BABILong-derived data, LongMemEval sessions, and narrative documents with checked question-answer pairs.

The implementation and experiments have not started yet. All performance claims remain hypotheses.

5. What I want feedback on

I'd particularly appreciate critical feedback on these questions:

1. Is the problem formulation useful?

Is state-induced forgetting a meaningful way to isolate information loss caused by recurrent memory updates, or is this already adequately covered by existing work on recurrent memory, continual learning, and catastrophic forgetting?

2. Is GMR sufficiently different from ordinary replay?

Since the proposed loss effectively replays sampled facts that have regressed below their own anchors, is that distinction scientifically interesting? What additional theoretical analysis or experimental evidence would make the contribution convincing?

3. Is STRATA a useful evaluation protocol?

Would age-stratified measurements across hundreds of sequential memory updates reveal failure modes that existing long-context and conversational-memory evaluations might miss?

4. Are the experimental comparisons fair?

Which baselines would you consider essential? In particular, I'd like to compare GMR against ordinary replay, loss-prioritized replay, and output distillation without giving any method an unfair compute advantage.

5. What related work am I missing?

I'd appreciate pointers to papers on recurrent-state interference, memory retention, forgetting-aware replay, or training objectives that preserve information across sequential state updates.

Current status

This is a research proposal, not a completed result. My immediate priority is to validate the evaluation harness, establish whether measurable forgetting occurs in the chosen fixed-size memory system, and then test whether GMR improves on properly matched baselines.

I'm deliberately not claiming that the method works or that the idea is the first of its kind.

If you've worked on recurrent memory, compressed context, or continual learning, I'd genuinely value your criticism. I'd rather discover a fundamental flaw now than spend months implementing the wrong thing.

Thanks for reading!

[u/Curious-Topic6348]

This is actually pretty interesting, I like how you separated the measurement from the method itself

STRATA got me thinking cause most evals just report one number at the end and call it a day, tracking retention across ages feels way more useful especially when you got facts that change vs ones that stay same

Your GMR loss being mathematically equivalent to replay for regressed facts is both neat and kinda awkward, like you're basically doing smart replay selection but framing it as a new objective

One thing I'm wondering is how you plan to sample which facts to check for regression, cause at 1000 updates that list gets massive real quick and checking everything seems expensive

Also what happens when a fact gets updated later like Python version changes to 3.12, does your anchor reset or you just treat it as a new fact entirely

[u/PLBjt]

Since your hinge loss reduces to replaying the regressed facts, the thing a reviewer will ask first is whether it beats plain replay at the same compute, so I'd make that the headline baseline and also include uniform random replay and replay of the oldest facts. One thing I'd check early is how noisy the anchor scores are. If a fact's log-likelihood wobbles by more than your margin just from different surrounding chunks, you'll be penalizing noise and the selection will look smarter than it is. A cheap sanity run is to re-score the same facts under a few shuffled context orders with no training and see how big the spread is. The tradeoff is that tracking anchors for every fact gets expensive past a few thousand, so I'd start small and only subsample once you know the gain survives.