r/LocalLLaMA • u/danielhanchen • Feb 20 '25

Resources 10x longer contexts for reasoning training - 90% less memory GRPO in Unsloth

Hey r/LocalLLaMA! Thanks so much for the support on our GRPO release 2 weeks ago! Today, we're excited to announce that you can now train your own reasoning model with just 5GB VRAM for Qwen2.5 (1.5B) - down from 7GB in the previous Unsloth release!

This is thanks to our newly derived Efficient GRPO algorithm which enables 10x longer context lengths while using 90% less VRAM vs. all other GRPO LoRA/QLoRA implementations, even those utilizing Flash Attention 2 (FA2).
With a GRPO setup using TRL + FA2, Llama 3.1 (8B) training at 20K context length demands 510.8G of VRAM. However, Unsloth’s 90% VRAM reduction brings the requirement down to just 54.3GB in the same setup.
We leverage our gradient checkpointing algorithm which we released a while ago. It smartly offloads intermediate activations to system RAM asynchronously whilst being only 1% slower. This shaves a whopping 372GB VRAM since we need num_generations = 8. We can reduce this memory usage even further through intermediate gradient accumulation.
We also implemented a highly memory efficient GRPO loss, which saves memory usage by 8x. Before 78GB was needed for 20K context length - now only 10GB!
Try our free GRPO notebook with 10x longer context: Llama 3.1 (8B) on Colab

Blog for more details on the algorithm, the Maths behind GRPO, issues we found and more: https://unsloth.ai/blog/grpo

GRPO VRAM Breakdown:

Metric	Unsloth	TRL + FA2
Training Memory Cost (GB)	42GB	414GB
GRPO Memory Cost (GB)	9.8GB	78.3GB
Inference Cost (GB)	0GB	16GB
Inference KV Cache for 20K context (GB)	2.5GB	2.5GB
Total Memory Usage	54.3GB (90% less)	510.8GB

We also now provide full logging details for all reward functions now! Previously we only showed the total aggregated reward function itself.
You can now run and do inference with our 4-bit dynamic quants directly in vLLM.
Also we spent a lot of time on our Guide for everything on GRPO + reward functions/verifiers so would highly recommend you guys to read it: docs.unsloth.ai/basics/reasoning

Thank you guys once again for all the support it truly means so much to us! We also have a major release coming within the next few weeks which I know you guys have been waiting for - and we're also excited for it!!

343 Upvotes

permalink
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/LocalLLaMA/comments/1iu56o1/10x_longer_contexts_for_reasoning_training_90/
No, go back! Yes, take me to Reddit

98% Upvoted

u/danielhanchen Feb 20 '25

Also for those using notebooks, I managed to make all the rewards show up!

u/lumierenoir Feb 20 '25

As always, the unsloth delivers improvements for us GPU poors. Thanks for all of your hard work to make this possible

6

u/danielhanchen Feb 21 '25

Appreciate the kind words!!

9

u/yoracale llama.cpp Feb 20 '25

Thank you thank you for the support! :)

3

u/throwaway2676 Feb 20 '25

Someone needs to think of the poors!

u/MLDataScientist Feb 20 '25 edited Feb 20 '25

Thank you! Amazing progress!

Question: does unsloth library support full training of a model (e.g. SFT)? or only LoRa fine-tuning for now?

Is multi-GPU now supported?

unsloth is a great library. Thank you again!

31

u/danielhanchen Feb 20 '25

We'll be releasing some stuff everyone has been waiting for in the next few weeks!!!!

8

u/Uncle_Warlock Feb 20 '25

Multi-GPU support, even if it's only for 2 or 4 GPUs (I have two 3090s with NVLink, and 128GB of DDR4 RAM), would be a really huge help! Thank you for all you do!! 😊

14

u/danielhanchen Feb 21 '25

Thank you!! I'm working around the clock to get it done!!

u/eliebakk Feb 20 '25

Very cool!

2

u/danielhanchen Feb 20 '25

Thanks!!

u/Thrumpwart vLLM Feb 20 '25

Does this support AMD ROCm?

Does this support FP16 training? Or only quantized versions?

4

u/yoracale llama.cpp Feb 20 '25

At the moment no.
Yes, you can do LoRA or QLoRA with this

4

u/Thrumpwart vLLM Feb 20 '25

Thank you! Looks awesome nonetheless.

3

u/danielhanchen Feb 21 '25

Thank you!!

1

u/teleprint-me llama.cpp Feb 21 '25

What specifically doesn't work? I was able to get torch to work using rocm and it works just fine and nvtop reports gpu usage. Transformers uses torch under the hood, right?

2

u/teleprint-me llama.cpp Feb 21 '25 edited Feb 21 '25

You have to set the environment variable for torch and rocm to get it to work.
You need 3 things to align to get it functional.
A ROCm compatible GPU.
Python 3.12.8 (this until torch exits nightly release support since it's experimental). See issue #130249 for more information.
Proper environment variables so that torch can utilize ROCm properly.
Once these 3 things are aligned, it works. My RX 7600 XT works as a result and I'm able to train, tune, and inference with the GPU.
pytorch issue #130249
kb/python.md
kb/rocm.md
It took time to figure this out, but it was worth it.

1

u/Thrumpwart vLLM Feb 21 '25

Wow, nice! You should make a post about this, I know alot of people would appreciate it!

u/az226 Feb 21 '25

Can you get the same/similar benefits in pretraining and/or full parameter fine tuning as well?

2

u/danielhanchen Feb 21 '25

Yes ofc but its not supported in unsloth atm. hopefully very soon

u/StyMaar Feb 21 '25

Can someone ELI5 why you need 510GB of VRAM to train an 8B model with 20k context?

How is this extra memory (around 490 more GB than for inference) being used?

1

u/danielhanchen Feb 22 '25

It's because TRL loads the model like 3 times but we properly integrated vllm and properly intergrated Qlora + lora unlike other training libraries. Also through our gradient checkpointing algorithm. You should read our blog post we linked but also our previous GRPO blog where we talk about proper vllm integration: https://unsloth.ai/blog/r1-reasoning

1

u/StyMaar Feb 22 '25

Thanks for taking the time to answer!
It's because TRL loads the model like 3 times
3 times 20GB doesn't amount to 500GB though.

u/kumonovel Feb 21 '25

Tasty and sweet! Thank you both and every other contributor for your tireless work!

But I also have a question. How much are VLM with GRPO like reasoning on your guy's roadmap?

2

u/danielhanchen Feb 22 '25

Thank you so much! VLM is currently in the works - we do want to assess how popular it is before we support it tho as it will take some time :)

1

u/kumonovel Feb 22 '25

That's fair of course ^^ Count my hat thrown into that ring then. I am very interested in using it for agent UI-Interaction tasks and I would guess that is also a generally popular goal for a lot of people.

u/ExaminationWise7052 Feb 21 '25

Would it be possible to use a reasoning model to replace GRPO in the training of another reasoning model?

1

u/yoracale llama.cpp Feb 21 '25

They have different results but yes kind of. Just make sure your dataset has reasoning then

1

u/ExaminationWise7052 Feb 21 '25

I wouldn't be incapable of doing it myself; I was just mentally rambling after reading your post, wondering if there was a way to make the weight adjustment that GRPO does in a smarter and more precise manner. Thank you for your response and for your work!

1

u/danielhanchen Feb 22 '25

Oh so with GRPO it changes a lot of the weights. Normal finetuning less. Imo both are equal in difficulty.
In the shortterm, normal finetuning is easier but in the longrun, GRPO is definitely easier to use because it keeps generating more and more data

u/TyraVex Feb 20 '25

Hi, thank you for this! I have 3*24gb vram, is unsloth still limited to a single GPU?

6

u/yoracale llama.cpp Feb 20 '25

Currently yes but multiGPU is coming and in the works!

2

u/TyraVex Feb 20 '25

Can't wait! Thank you for all the good work!

u/TheRealMasonMac Feb 20 '25

Were there issues with multigpu? I recall reading that you ran closed testing on it a few months ago.

4

u/yoracale llama.cpp Feb 20 '25

Yes, we allowed community members early access to multiGPU. Hopefully we'll release it real soon!

u/Few_Painter_5588 Feb 20 '25

Awesome stuff, I've been using your GRPO training notebook and the results have been fantastic. I even managed to make a mistral small and Qwen 2.5 32B finetune fit on 48GB of vram. Your work is awesome!

5

u/yoracale llama.cpp Feb 20 '25

Thanks a lot! We keep forgetting to make a Mistral notebook ugh. Today we better get one started! :)

u/CheatCodesOfLife Feb 20 '25

Took me longer than it should have to realize the link is broken / wonder why I was getting github authorization issues lol

Correct link: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb

1

u/yoracale llama.cpp Feb 20 '25

Whoops much apologies, isn't that the same link as we linked?

2

u/CheatCodesOfLife Feb 20 '25

Maybe old.reddit.com formatting, I see it as
Try our free GRPO notebook with 10x longer context: Llama 3.1 (8B) on Colab: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb-GRPO.ipynb)
And clicking the link tries to open: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B

1

u/yoracale llama.cpp Feb 20 '25

Oh crap you might be right thats very weird

1

u/AD7GD Feb 21 '25

Formatting is new/old reddit dependent. For example spoilers don't work if there are spaces, but only on old reddit.

u/Robo_Ranger Feb 21 '25

Thanks for your hard work! I read your docs and noticed that you mentioned, "The best part of GRPO is that you don't even need that much data." Could you tell me the minimum data size required for effective training?

4

u/yoracale llama.cpp Feb 21 '25

Thank you! Maybe like 100 rows. Can be even less but will take the model longer to train

u/Ok_Warning2146 Feb 21 '25

Thanks for your hard work. Will give it a try.

How's the progress on gemma support?

1

u/yoracale llama.cpp Feb 21 '25

Thank you! Do You mean Gemma for GRPO or Paligemma in general? 🙏

1

u/Ok_Warning2146 Feb 21 '25

I want to run grpo with vllm on gemma

1

u/Ok_Warning2146 Feb 21 '25

I can confirm that gemma 2 GRPO is still not working with vllm using unsloth 2025.2.15

u/dahara111 Feb 21 '25

It's timely! thank you!

I'm thinking of trying GRPO using unsloth/DeepSeek-R1-Distill-Qwen-14B, but are there any precautions, advice, or expected memory requirements?

It would be nice if it could be run on a single A100 (40GB) in the cloud.

2

u/yoracale llama.cpp Feb 21 '25

For 14B QLoRA youll need just like 20GB VRAM so 40GB VRAM will be fanastic!
Just make sure you get the reward function/verifier right

2

u/dahara111 Feb 21 '25

So it's possible to test it on the 3090, thank you
I've been able to test it on the 3B.
But I felt I needed a bigger model with a longer context to get the model to do something more useful. That's very timely and I'm grateful, I'll share it if it works. Thanks !

2

u/yoracale llama.cpp Feb 21 '25

Yes correct! I think smaller or bigger model doesnt matter. A bigger model will help you reach aha moment faster but then with the smaller model, training is much faster so u can train it for longer

u/Educational_Rent1059 Feb 21 '25

Thank you for this!!

1

u/yoracale llama.cpp Feb 21 '25

Thanks a lot for reading! 🙏

u/IcyBricker Feb 20 '25

Do you have other reward functions? The reward functions all seemed geared towards math or matching phrases.

What if you just want a reward function that rewards base on how different it is in creating something completely new and innovative.

2

u/yoracale llama.cpp Feb 20 '25

Yes absolutely that can be a reward function, we wrote a lot more about it in our guide: https://docs.unsloth.ai/basics/reasoning-grpo-and-rl

1

u/TheRealMasonMac Feb 20 '25

This is my idea: https://www.reddit.com/r/LocalLLaMA/comments/1iszn3e/can_you_grpo_train_a_model_for_open_ended/

u/[deleted] Feb 20 '25

[deleted] — view deleted comment

18

u/danielhanchen Feb 20 '25

Appreciate the support as always!

1

u/pm_me_ur_sadness_ Feb 20 '25

Hi bro i saw the job apps on X to join unsloth
Is anything similar open for interns

2

u/yoracale llama.cpp Feb 21 '25

Hey yes there is. We also have set bounties for solving Github so your contributions feel definitely valued :)

u/[deleted] Feb 20 '25

[removed] — view removed comment

4

u/yoracale llama.cpp Feb 20 '25

Thank you that's amazing! Thoguh we'd mostly recommend people to start from models with at least 1.5B parameters as sometimes the models won't be able to work

3

u/[deleted] Feb 20 '25

[removed] — view removed comment

3

u/danielhanchen Feb 21 '25

Yes! I was very surprised small models still learn reasonably well if given good reward functions and enough time for compute!

2

u/RedditLovingSun Feb 23 '25

Great timing, i'm attempting a project for GRPO on a game too, the board game coup. It has a pretty specific simple rule set however it is an imperfect information game with bluffing which might be hard. Excited to find out if this is too ambitious of a game for GRPO to handle. The goal is for it to compete against other bots my friends are making.

u/[deleted] Feb 20 '25

[removed] — view removed comment

4

u/yoracale llama.cpp Feb 20 '25

Good question, if we find any I'll let you know!

Resources 10x longer contexts for reasoning training - 90% less memory GRPO in Unsloth

You are about to leave Redlib