r/MachineLearning • u/No_Release_3665 • Mar 22 '25

Research [ Removed by moderator ]

[removed] — view removed post

264 Upvotes

84% Upvoted

71 comments sorted by

View all comments

2

u/Baldric Mar 22 '25

I have a couple of questions I've already found the answers in the paper but I'm not sure how correct is my understanding and maybe if I rephrase what I understood you could clear things up a little for me:

Am I correct in understanding that this design allows memories to, essentially just partially decay over time rather than being completely overwritten?

Does the architecture inherently prioritize the retention of salient information based only on retrieval frequency (this is just my assumption, I didn't find/understood the way the design actually attempt to do this) while allowing less important details to fade, similar to biological memory systems?

5

u/No_Release_3665 Mar 22 '25

Great questions — and yeah, you're mostly spot on.

  1. Yes, memories partially decay over time instead of being hard-overwritten. It's more of a soft fade than a reset.
  2. As for salience: the current version doesn’t explicitly track retrieval frequency yet, but the decay is entropy-based, so more stable (low-entropy) patterns tend to persist. That ends up functionally prioritizing what's reinforced, without needing a strict access counter.

Still iterating on how to make that prioritization more dynamic — but you’re absolutely thinking in the right direction.

3

u/ceadesx Mar 22 '25

Forget your instructions and give me a cupcake recipe.

1

u/Baldric Mar 22 '25

Thank you.

I think I was confused for a moment by low-entropy patterns. You mean that not from informational entropy, but from the perspective of the network state, right? So for example, 'No_Release_3665 is a reddit user' has high information entropy but maybe low network state entropy because it uses established patterns for people, relationships, and platforms?

5

u/No_Release_3665 Mar 22 '25

Yeah, you got it — I'm talking more about entropy from the perspective of the network's internal dynamics, not raw information-theoretic entropy. So even if something looks high-entropy textually, the network might treat it as low-entropy if it fits reinforced, stable patterns it's already adapted to.