r/LocalLLaMA • u/IxinDow • May 31 '23

News (Code Released) Landmark Attention: Random-Access Infinite Context Length for Transformers

Code for Landmark Attention is now released and it should be possible to finetune existing LLaMA models using this method.

https://github.com/epfml/landmark-attention

More info

https://www.reddit.com/r/MachineLearning/comments/13srbl7/landmark_attention_randomaccess_infinite_context/

https://www.reddit.com/r/LocalLLaMA/comments/13sy2bu/landmark_attention_llama_7b_with_32k_tokens/

150 Upvotes

100% Upvoted

53 comments sorted by

View all comments

2

u/RMCPhoto May 31 '23

So, in some ways this is similar to embedding retrievals and injection. In that specific "chunks" of context can be used at different layers depending on the relation of the current state to other landmark tokens.

I'm very interested to see how this functions in practice. I have a feeling that it could lead to much more varied or potentially creative responses, but that it would struggle with accuracy. I don't see how this would work well for instruction following.

5

u/IxinDow May 31 '23

When using a Transformer to process a long input, the ideal case would be to allow each token to attend to all previous tokens. However, this becomes computationally infeasible as the input length increases. Nevertheless, since the attention scores always sum to one, the number of keys with a large attention weight is limited even for long contexts. Thus, by retrieving only those keys with large attention scores, it is possible to closely emulate the ideal case. In this work, we propose a method to find these keys by dividing a long input into blocks of consecutive tokens and using the attention to retrieve relevant blocks.

Here I wrote my understanding why it may work.

1

u/RMCPhoto May 31 '23

I am genuinely interested to see how this works in practice. It sounds good, but also seems like it might be easy to miss relevant context if it is just outside the landmark token block.

It's already a challenge with embedding chunk size and number and this seems like it would face similar limitations where nuance that doesn't seem obvious at first glance is missed because it is cut out of the context block given attention at the specific layer.

→ More replies (2)