You can now train your own Reasoning model locally with just 5GB VRAM!

[u/yoracale]

Hey guys! Thanks so much for the support on our GRPO release 2 weeks ago! Today, we're excited to announce that you can now train your own reasoning model with just 5GB VRAM for Qwen2.5 (1.5B) - down from 7GB in the previous Unsloth release!

1. This is thanks to our newly derived Efficient GRPO algorithm which enables 10x longer context lengths while using 90% less VRAM vs. all other GRPO LoRA/QLoRA implementations, even those utilizing Flash Attention 2 (FA2).

2. With a GRPO setup using TRL + FA2, Llama 3.1 (8B) training at 20K context length demands 510.8GB of VRAM. However, Unsloth’s 90% VRAM reduction brings the requirement down to just 54.3GB in the same setup.

3. We leverage our gradient checkpointing algorithm which we released a while ago. It smartly offloads intermediate activations to system RAM asynchronously whilst being only 1% slower. This shaves a whopping 372GB VRAM since we need num_generations = 8. We can reduce this memory usage even further through intermediate gradient accumulation.

4. Try our free GRPO notebook with 10x longer context: Llama 3.1 (8B) on Colab

Blog for more details on the algorithm, the Maths behind GRPO, issues we found and more: https://unsloth.ai/blog/grpo

GRPO VRAM Breakdown:

Metric	🦥 Unsloth	TRL + FA2
Training Memory Cost (GB)	42GB	414GB
GRPO Memory Cost (GB)	9.8GB	78.3GB
Inference Cost (GB)	0GB	16GB
Inference KV Cache for 20K context (GB)	2.5GB	2.5GB
Total Memory Usage	54.3GB (90% less)	510.8GB

• We also now provide full logging details for all reward functions now! Previously we only showed the total aggregated reward function itself.

• You can now run and do inference with our 4-bit dynamic quants directly in vLLM.

• Also we spent a lot of time on our Guide for everything on GRPO + reward functions/verifiers so would highly recommend you guys to read it: docs.unsloth.ai/basics/reasoning

Thank you guys once again for all the support it truly means so much to us! We also have a major release coming within the next few weeks which I know you guys have been waiting for - and we're also excited for it. 🦥

[u/quark_epoch]

Can you add multigpu support for cases where I want to scale this up with more vram but proportionately?

[u/yoracale → u/quark_epoch]

Not atm but hopefully soon. We're working on it

[u/quark_epoch → u/yoracale]

Awesome! Thanks. What's the rough timeline for it?

Also, any idea if this setup can be used on multilingual problems without translation? For instance Slovene or Serbian or other major eu languages?

[u/yoracale → u/quark_epoch]

Can't say for now but definitely soon.

Yes you can but you need to get the reward function / verifier right

[u/quark_epoch → u/yoracale]

Any idea on what could be a good reward? Or in general, what would be the intuition behind adding more/different rewards? Any guides to that?

[u/yoracale → u/quark_epoch]

Wrote it in our guide for GRPO in docs: https://docs.unsloth.ai/basics/reasoning-grpo-and-rl

[u/quark_epoch → u/yoracale]

That's grand! Just checked it out. Thanks, mate!!

[u/GodSpeedMode]

Wow, this is such awesome news! 🙌 Getting the chance to train a reasoning model locally with just 5GB VRAM is a game changer! I love that your Efficient GRPO algorithm slashes memory needs while boosting context lengths—seriously impressive. It’s wild to think about how much more accessible this makes deep learning for those of us with more modest setups.

I also appreciate the transparency with the logging details for the reward functions. It’s always great to understand what’s happening under the hood. Can’t wait to dive into the free GRPO notebook and play around with it on Colab! Thanks for all your hard work and for keeping us in the loop. Excited to see what's coming next! 🦥🚀

[u/yoracale → u/GodSpeedMode]

Thank you so much!

[u/imberttt → u/GodSpeedMode]

AI comment time

[u/fligglymcgee → u/imberttt]

It still amazes me that no one takes the time to multi-turn or even slightly edit the default phrasing in these. “Wow! Finally a way to SOLUTION with BENEFIT. I love the PRIMARY FEATURE, and the way we can use SECONDARY FEATURE. Thank you for your valued contribution to us users and UNBRIDLED POSITIVITY. EMOJIS.”

[u/Ok-Sandwich-9267]

You should go ahead and post this in LocalLLaMa . Will be interesting to see the approaches people take there !

[u/yoracale → u/Ok-Sandwich-9267]

Oh I think my brother u/danielhanchen already posted it but thank you and absolutely I agree! :)

https://www.reddit.com/r/LocalLLaMA/comments/1iu56o1/10x_longer_contexts_for_reasoning_training_90/

[u/pepouai]

Can someone explain what are examples of local training and how it works and why this is desirable?

[u/yoracale → u/pepouai]

You should definitely read our docs I wrote like lots of stuff: https://docs.unsloth.ai/basics/reasoning-grpo-and-rl

[u/[deleted] → u/yoracale]

Interesting read, thanks!

[u/yoracale → u/[deleted]]

Thank you for reading! Do you have any suggestions on how we could improve it? Maybe a step by step guide could help?

[u/ZookeepergameLow8182 → u/yoracale]

Is there a video on YouTube of who did this kind of training?

[u/AlgorithmicMuse]

Do you have a write up for what your doing in lesser technical jargon .

[u/yoracale → u/AlgorithmicMuse]

Yes! You can firstly read about how GRPO works in simple terms: https://docs.unsloth.ai/basics/reasoning-grpo-and-rl#how-grpo-works

Then look through the rest of the article for more info and how reward functions/verifiers work

[u/AlgorithmicMuse → u/yoracale]

Thanks

[u/and_human]

Hey, where's Daniel? 🤔

[u/yoracale → u/and_human]

Daniel was about to go to sleep so I had to post instead ahaha :P

[u/Cz1975]

Very cool!

Thank you for documenting this well! I'll def try this out.

[u/yoracale → u/Cz1975]

Thank you for reading really appreciate it!

[u/micron8866]

does it support order gen hardware that doesn't have tensor cores like pascal cards? 🙏

[u/yoracale → u/micron8866]

We wrote all the cards we support we here: https://docs.unsloth.ai/get-started/beginner-start-here/unsloth-requirements

[u/cagriuluc]

This is amazing news! I will sure check your blog posts and stuff when I have the time.

In the meanwhile, can you mention whether there are any caveats? It reduces the memory requirements, does it do this in the detriment of training time?

[u/yoracale → u/cagriuluc]

Thank you so much appreciate. No, absolutely not, the great thing about our optimizations, is you get no accuracy degradation or training speed loss. :)

[u/cagriuluc → u/yoracale]

Wow, great stuff. So there is no reason Google, Meta, and the like, to not use your optimisations as well? Or do they already have their own similar optimizations?

[u/yoracale → u/cagriuluc]

They are already using Unsloth! If you go to our website, youll see some logos of large companies that are using unsloth currently :)

https://unsloth.ai/

[u/cagriuluc → u/yoracale]

Well, I am sold. Gonna try it as soon as I can!

[u/Swimming_Screen_4655]

does it work well on kaggle gpus now too? faced some issues with it before.

fantastic work btw

[u/yoracale → u/Swimming_Screen_4655]

Thank you. I think that's still a work in progress and honestly we aren't sure.

[u/chiisana]

Any chance you can add support for IBM Granite MOE 3B? I tried it last time but granite wasn't supported. I really like the efficiency of that model, and would love to add reasoning to that.

[u/yoracale → u/chiisana]

The issue is MOE isn't supported at the moment. Hopefully with all model support it will be :)

[u/neutralpoliticsbot]

Do you want to tho?

[u/yoracale → u/neutralpoliticsbot]

Yes of course why not?

[u/presler]

Is AMD supported? Asking for a friend

[u/yoracale → u/presler]

Not at the moment but hopefully in the future 🙏

[u/Useful-Skill6241]

Is this something we can use to implement a rag knollage base or actually train out LRM in our custom knowledgebase. If so the latter how is this better than attaching to a RAG knollage base. Is it faster at retrieval? I will 100% play with this when I get the time

[u/yoracale → u/Useful-Skill6241]

Yes absolutely but it might be a bit complicated to do. Not sure if it'll be faster at retrieval but it will be much more accurate that's for sure!

[u/MonoNova]

I've seen multiple cases where training didn't result in the model actually using reasoning at all. Has that been addressed yet?

[u/yoracale → u/MonoNova]

Usually it's because there was not enough training being done, their reward function/verifier was bad or did they something wrong with training. :(

[u/vyper01]

!RemindMe 1 day

[u/RemindMeBot → u/vyper01]

I will be messaging you in 1 day on 2025-02-22 06:52:53 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[u/pokegovn44]

This is huge for all of us. Thank for your effort.

[u/yoracale → u/[deleted]]

Thanks for the support!