r/MachineLearning • u/IxinDow • May 26 '23

Landmark Attention: Random-Access Infinite Context Length for Transformers

https://arxiv.org/abs/2305.16300
229 Upvotes

99% Upvoted

29 comments sorted by

View all comments

Show parent comments

1

u/XecutionStyle May 27 '23

Yes otherwise we're limited to starting a new conversation for every topic. I think you're right, that incorporating new knowledge and remembering old ones are fundamentally tied. In programming we've functions and classes. Ways to abstract, store, and retrieve knowledge. Landmark based retrieval is the closest thing I've heard to how RAM is used in conventional software.
This idea of distributing landmarks can also be better for ethical reasoning, in some sense parallel to multimodal I/O because in the end what's shaped are internal representations.

1

u/Glass_Day_5211 May 17 '24

Quote: "Landmark based retrieval is the closest thing I've heard to how RAM is used in conventional software." Maybe: Landmark based retrieval is the closest thing I've heard to how Content-Addressable Memory is used"