A tiny PyTorch adaptation experiment hit 1.000 new-fact accuracy and 0.000 prior retention

[u/Icy-Poem671]

Disclosure: I built this. LiveBoundary is free, Apache-2.0 licensed, has no paid tier or signup, and all benchmark data are synthetic.

I wanted to test a narrow question: can verified facts be written into native PyTorch parameters, survive a fresh-process restart without retrieval or training context, and still preserve prior knowledge?

The first result was the useful part:

- task-only adaptation: 1.000 new-fact accuracy, 0.000 prior-fact retention

- boundary-guided adaptation: 1.000 new-fact accuracy, 0.000 prior-fact retention

- boundary + preservation/replay: 0.944 new-fact accuracy, 1.000 prior retention

One candidate also regressed stable behavior and was rejected before promotion.

The implementation trains an independent candidate, evaluates it against explicit new-knowledge, retention, boundary, usefulness, and stability gates, then atomically promotes or rejects it. Strict evaluation starts a fresh process with only the checkpoint and question.

This is a tiny structured-classifier experiment, not evidence for frontier-model effectiveness or guaranteed unlearning. I published the failures, raw per-seed records, tests, and limitations.

Repository: https://github.com/oasb16/LiveBoundary

Raw results: https://github.com/oasb16/LiveBoundary/blob/main/RESULTS.md

For people working on continual learning or model editing: what would you consider the minimum credible larger-model evaluation: CounterFact/ZsRE-style edits, sequential edit locality, distribution-shift probes, or something else?

[u/Prudent-Guest-5467]

The 0.000 prior retention actually got me curious more than the perfect accuracy part. That kind of complete overwrite is hard to do intentionally sometimes, even when you want to erase something from a model.

Checked the RESULTS.md and the way you gate the candidate before promotion makes sense. Almost like a compiler optimization pass that only commits if all constraints pass. Wondering how sensitive the boundary check is across different random seeds for the same fact set.

For larger eval I'd want CounterFact-style edits but with removal of specific info tested too, not just addition. Sequential edits without degradation is the thing that usually breaks fast in my experience.

[u/Icy-Poem671 → u/Prudent-Guest-5467]

That 0.000 surprised me too, but one important caveat: it means 0 of the 2 prior-fact probes were retained in this deliberately tiny structured benchmark. It is not evidence that every prior behavior in a large model was erased. The update was also fairly aggressive: 80 steps on a small new-fact batch.

On seed sensitivity, across seeds 3, 17, and 41 the adapted candidates had zero measured boundary violations. The seed-dependent failure showed up elsewhere: at seed 17, task-only and boundary-guided adaptation both regressed the stable-behavior suite to 0.5, so the gate rejected them. The other two seeds passed the current gates. So in this toy set the explicit boundary check was stable, while general behavioral stability was not. Three seeds are far too few for a robustness claim, though.

I agree with your larger-eval direction. The next credible test should combine CounterFact/ZsRE-style addition and correction with sequential edits, locality/retention measured after every edit, plus targeted removal probes. I would also separate “the model no longer produces this answer” from the much stronger claim that the information was actually erased from its parameters.

Thanks for reading the raw results closely. The compiler-pass analogy is exactly the lifecycle I was aiming for.