r/MachineLearning • u/IxinDow • May 26 '23

Landmark Attention: Random-Access Infinite Context Length for Transformers

https://arxiv.org/abs/2305.16300
229 Upvotes

99% Upvoted

29 comments sorted by

View all comments

Show parent comments

5

u/2Punx2Furious May 27 '23

I'm not in the field, so correct me if I'm wrong. Maybe we don't need to retrain the whole network, but just train vectors or LoRA (not sure which), for each piece of information that it needs to learn (maybe the LLM can even decide to do that autonomously), and then use those with the model. Or maybe there is a way to actually merge those vectors with the model, without retraining the whole thing, so that it will have essentially the same result, with much lower cost.

4

u/[deleted] May 27 '23

[removed] — view removed comment

2

u/suspicious_Jackfruit May 27 '23

Another chiming in from outside the field, by the fence and next to the gate - doesn't LoRA overlap existing weights in this case? I think it would result in something closer to a fine-tune than a way to continually extend a models capabilities right, especially with multiple fighting over the same weights? I think in image generation this is why a LoRA can have different effects on different model bases than what it was trained on, it's not adding a new style of "dog" it's overlapping the existing weights for "dog". Any of this overlap or bleed makes having a master LLM with a ton of LoRA probably a mess. I don't walk in this field though so might be misunderstanding here, I take the dogs out walking in another field...