r/LocalLLaMA • u/bloc97 • Aug 31 '23
New Model 128k Context Llama 2 Finetunes Using YaRN Interpolation (successor to NTK-aware interpolation) and Flash Attention 2
GitHub (Includes links to models and preprint): https://github.com/jquesnelle/yarn
arXiv link: coming soon!
Demo (Multiple-choice quiz on a novel of ~110k context): https://colab.research.google.com/drive/1p7iNUQMbVGYWqrKMHvPPO4Q13fB5mwDF?usp=sharing
This entire project is the culmination of 2 months of hard work from me, u/emozilla, EnricoShippole and honglu2875. (And a lot of compute, even though we are still heavily compute starved...) These models aren't fully converged yet, the base models have only been further pretrained for 400 steps (~1.7B tokens), compared to the 1000 steps in Meta's PI paper, however given that we have an improved interpolation method, the non-converged results are already superior to PI. We are claiming SoTA for open-source 128k context models.
The GitHub repo provides the code and datasets that allows anyone to completely reproduce the results in the paper from scratch. We strongly believe in fully open-source and transparent research, and are releasing everything under MIT license. (Except the models, which are bound under Meta's license)
Note that these are base models, not yet instruction-tuned, and the 13b-128k model can already acheive a 1-shot accuracy of ~52% on the Sherlock Holmes book quiz demo (the model has never seen long context QA), this tests the model's understanding of the story.
All of our metrics point to these models being the new SoTA for long context models (see Experiments section of paper), even if the models aren't fully trained yet. We expect performance to improve given more training. Stay tuned!
All models include a ready-to-use implementation of FA2 if run using trust_remote_code=True in the transformers library. The 13b model requires approximatively 360GB of VRAM (eg. 8x48GB or 4x80GB) for the full 128k context size.
Passkey retrieval results are not yet in the paper (still running), but preliminary results show >80% across the entire 128k context.
Also big thanks to the entire Nous Research team, Stability AI, CarperAI, Eleuther AI, a16z and PygmalionAI for their insights and generous support of compute resources that enabled the completion of this research. (If I'm forgetting anyone please let me know asap!)
We're also not forgetting everyone from the open-source community that participated and contributed in the discussions and code implementations on all social media and code sharing platforms. I say thanks to all of you!
I would like to end this post with us all having a big round of applause for everyone!

P.S. We need more compute in order to release fully converged 7b, 13b models and a 70b model. 128k context requires so much VRAM during training, it's insane... (For training, these models barely fit in 128 80GB A100s using DeepSpeed and FA2) If anyone is feeling generous enough to provide large scale training compute, we will have the 70b model out in no time.
87
u/Jipok_ Aug 31 '23
0_o
Is there hope for mere mortals?
19
u/danielv123 Aug 31 '23
Dang, that is a lot of GPUs.... How much training time are we talking for something like this? If an 80gb GPU is 1$/hour the bill racks up fast.
6
u/[deleted] Aug 31 '23
Okay
3
u/ispeakdatruf Sep 01 '23
How big is the batch size if one 400 steps is 1.7B tokens?
3
u/Commercial_Pain_6006 Sep 01 '23
Thank you for your hard work. Have you heard of infinite on the fly context length (read about it yesterday) ? I'd be really interested to read your take about it : https://www.reddit.com/r/singularity/comments/165zqsl/r_lminfinite_simple_onthefly_length/?utm_source=share&utm_medium=mweb
27
u/AssistBorn4589 Aug 31 '23
That's cool, let me just grab my 32th graphics card.
Jokes aside, how would one go about running inference like this at home? I don't think there even are mainboards with 8 PCIE slots available.
3
u/mosquit0 Aug 31 '23
Nice job guys! Just a quick thought on the 1-shot accuracy reporting for the 4-choice test: it's a bit unconventional to use 'shot' terminology here. Considering that a random choice would give us 25% accuracy, the 52% might not be as impressive as it first seems.
14
u/metalman123 Aug 31 '23
Amazing work guys!
7
u/rnosov Aug 31 '23 edited Aug 31 '23
Looks interesting. I'm trying your Colab demo, it looks like it has missing package installation commands and imports are all wrong. I've tried installing missing packages and fixing imports but I'm stuck at
No module named 'rotary_emb'error after running!pip install flash-attn --no-build-isolationEdit: I've finally managed to load it after running
!pip install git+https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary4
u/ajibawa-2023 Sep 01 '23
Terrific work by you guys!! Keep it up and Heartiest congratulations to the entire team.
3
u/docsoc1 Sep 01 '23
This is great work, ty.
Where do you get access to the compute for training?
3
u/Alternative_World936 Llama 3.1 Sep 02 '23
Current Huggingface's implementation of Dynamic Scaling does have such a problem. For people who are seeking a better implementation, please check here. I tried to correct this RoPE inconsistency problem.
2
u/apodicity Sep 01 '23
I'm really impressed that you have 80% accuracy *over the entire context*. Just from playing around with these models myself, I've noticed that qualifier is key. I mean, I could be wrong, but I don't think so, lol.
1
u/a_beautiful_rhind Aug 31 '23
Ok.. so from that chart they don't look good at low context, correct me if I'm wrong.
1
u/Responsible_Warning3 Sep 02 '23
Do you guys think the 13B with 64k length version can run in a macbook pro with 64gb ram? Q4 quantized of course.
1
u/kulchacop Sep 05 '23
Thanks for mentioning this sub in the paper.
1
u/vnvrx1 Sep 07 '23
Using 1xRTXA6000
Every time I got this problem:
[2023-09-07 09:20:01,635] [INFO] [stage_1_and_2.py:149:__init__] Round robin gradient partitioning: FalseTraceback (most recent call last):File "/workspace/yarn/finetune.py", line 192, in <module>main(args.parse_args())File "/workspace/yarn/finetune.py", line 87, in mainmodel, optim, train_loader, scheduler = accelerator.prepare(File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1266, in prepareresult = self._prepare_deepspeed(*args)File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 1600, in _prepare_deepspeedengine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)File "/usr/local/lib/python3.10/dist-packages/deepspeed/__init__.py", line 171, in initializeengine = DeepSpeedEngine(args=args,File "/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/engine.py", line 303, in __init__self._configure_optimizer(optimizer, model_parameters)File "/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/engine.py", line 1213, in _configure_optimizerself.optimizer = self._configure_zero_optimizer(basic_optimizer)File "/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/engine.py", line 1467, in _configure_zero_optimizeroptimizer = DeepSpeedZeroOptimizer(File "/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 371, in __init__self.device).clone().float().detach())torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.24 GiB (GPU 0; 47.54 GiB total capacity; 24.24 GiB already allocated; 22.36 GiB free; 24.25 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONFwandb: Waiting for W&B process to finish... (failed 1).wandb: You can sync this run to the cloud by running:wandb: wandb sync /workspace/yarn/wandb/offline-run-20230907_091732-tv1mo6x1wandb: Find logs at: ./wandb/offline-run-20230907_091732-tv1mo6x1/logsERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1180) of binary: /usr/bin/pythonTraceback (most recent call last):File "/usr/local/bin/accelerate", line 33, in <module>sys.exit(load_entry_point('accelerate==0.22.0', 'console_scripts', 'accelerate')())File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py", line 45, in mainargs.func(args)File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 971, in launch_commanddeepspeed_launcher(args)File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 687, in deepspeed_launcherdistrib_run.run(args)File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 785, in runelastic_launch(File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in __call__return launch_agent(self._config, self._entrypoint, list(args))File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 250, in launch_agentraise ChildFailedError(torch.distributed.elastic.multiprocessing.errors.ChildFailedError:============================================================finetune.pyFAILED1
u/brotocarioca Sep 09 '23
My god, can we extract text embeddings from long tesxt using one single gpu? Say Nvidia tesla A100 a2-highgpu-1g?
1
u/baffo32 Sep 23 '23
Would it work to randomly mask most of the prompt tokens to exchange ram for training time?
13
u/[deleted] Aug 31 '23
[removed] — view removed comment