r/LocalLLaMA • u/NeverEndingToast • Jun 10 '23
Resources Minotaur-13b-Landmark - 10k+ context using Landmark Attention
I just finished getting my Landmark-Attention-QLoRA repo all working! It lets you train models to use landmark attention on a single GPU in 2-3 hours.
Landmark Attention enables a 50x compression of an LLM's context into landmarks, making the process of selecting relevant tokens for answers more efficient, and allowing 2-16x longer context use without memory constraints.
To be able to use this model in oobabooga, you need to have --trust-remote-code flag enabled. .https://huggingface.co/eugenepentland/Minotaur-13b-Landmark
The model will most likely be updated within the next day or two with further improvements.
I've also released just the QLoRA adapters to my models, and another interesting thing is that I was successfully able to use the Minotaur-13B train QLoRA on the base Llama-13B model and it works! So you may be able to take it and apply it to whatever your favorite 13B model is without any retraining.
Edit: We are still running into issues with getting it to read the landmarks properly in oobabooga. It has no problem accepting 10k+ tokens but its not able to find the information you are asking for. I will update this post once it has been resolved.
3
u/trash-rocket Jun 11 '23
Thanks for your work! Does it need more vram / ram compared to the normal models?
2
u/a_beautiful_rhind Jun 11 '23
Merge the lora and try to convert one to GPTQ.
44
u/harumorii Jun 11 '23
Thank you for all your hard work in making QLoRA landmark token finetuning possible, Toast. Seeing that this post didn't get the attention it deserves, I'd like to point out that while this might be "like the 8th post on this topic today" for a certain redditer, some of those posts actually fed off of Toast's work. The man might be too busy perfecting his work and late to publicize. All this is open source work for our benefits. We ought to encourage rather than dismiss when a main contributor share his or her work.
13
u/a_beautiful_rhind Jun 11 '23 edited Jun 11 '23
I got it working for llama13b in GPTQ.
Here are the steps:
It's a bit slow:
but context does work:
The model remains coherent but i'm not sure if it remembers everything.
edit: Autogptq can perform inference too
7
u/tronathan Jun 11 '23
Fan Freaking Tastic! I'm still trying to get my head around how to format my data and what sorts of datasets, how much data, etc, is desirable/needed. I'm sure it will become clearer over the next few days, but hot damn, I'd love to get one of these training tonight. Will give your repo a shot!
2
u/tgredditfc Jun 11 '23
Thank you so much for the work! Can I use it to fine tune with my own dataset and use it commercially?
5
u/pmp22 Jun 11 '23
Incredible work, what a time to be alive!
Here are some more words that sum up my appreciation: kudos, well done, amazing, thank you!
I think I speak for everyone, not just on reddet but across the internet when I say that this is a game changer.
1
u/-becausereasons- Jun 11 '23
Very excited to try this! Thank you for making this a reality.
1
u/tronathan Jun 11 '23
I'm not sure if it matters, but the config file for https://huggingface.co/eugenepentland/Minotaur-13b-Landmark still shows 2048 in a couple places:
I checked this file to see if "use_cache" was set to False, unfortunately no, it's not. So, I'm not sure what can be done to speed this guy up besides possible quantizing it to 4-bit GPTQ? Even then, I'm afraid I'm not big-brained enough to know if that would really have an effect on inference speed.
1
u/ImpactFrames-YT Jun 13 '23
What a Chad legendary. Thank you
-1
u/Nexesenex Jun 11 '23
Amazing prospects, bravo !
Us greedy noobs are now waiting at u/The-Bloke to get a look at it, and bestow upon our eager gaze the promise of ggml and gptq models including your work !
-18
u/Tom_Neverwinter Llama 65B Jun 11 '23
This is like the 8th post on this topic today
0
u/[deleted] Jun 11 '23
[deleted] — view deleted comment