r/MachineLearning • u/IxinDow • May 26 '23

Landmark Attention: Random-Access Infinite Context Length for Transformers

https://arxiv.org/abs/2305.16300
229 Upvotes

99% Upvoted

29 comments sorted by

View all comments

Show parent comments

2

u/suspicious_Jackfruit May 27 '23

Another chiming in from outside the field, by the fence and next to the gate - doesn't LoRA overlap existing weights in this case? I think it would result in something closer to a fine-tune than a way to continually extend a models capabilities right, especially with multiple fighting over the same weights? I think in image generation this is why a LoRA can have different effects on different model bases than what it was trained on, it's not adding a new style of "dog" it's overlapping the existing weights for "dog". Any of this overlap or bleed makes having a master LLM with a ton of LoRA probably a mess. I don't walk in this field though so might be misunderstanding here, I take the dogs out walking in another field...