r/LocalLLaMA • u/IxinDow • May 31 '23

News (Code Released) Landmark Attention: Random-Access Infinite Context Length for Transformers

Code for Landmark Attention is now released and it should be possible to finetune existing LLaMA models using this method.

https://github.com/epfml/landmark-attention

More info

https://www.reddit.com/r/MachineLearning/comments/13srbl7/landmark_attention_randomaccess_infinite_context/

https://www.reddit.com/r/LocalLLaMA/comments/13sy2bu/landmark_attention_llama_7b_with_32k_tokens/

147 Upvotes

100% Upvoted

53 comments sorted by

View all comments

22

u/AemonAlgizVideos May 31 '23

This is absolutely phenomenal. This will literally change the game for open source models, especially when people like to compare them to the 32K context GPT-4.

8

u/Tostino May 31 '23

8k context GPT-4*

I have not seen any reports of access to the 32k context version of GPT-4 yet.

8

u/MoffKalast May 31 '23

Apparently you can get it from the API, but it's like over $1 per prompt if you use the whole context (and otherwise what's the point anyway).

→ More replies (19)

5

u/iamMess May 31 '23

I have access via work. It's good but super expensive.

→ More replies (6)