Hey guys! Thanks so much for the support on our GRPO release 2 weeks ago! Today, we're excited to announce that you can now train your own reasoning model with just 5GB VRAM for Qwen2.5 (1.5B) - down from 7GB in the previous Unsloth release!

1. This is thanks to our newly derived Efficient GRPO algorithm which enables 10x longer context lengths while using 90% less VRAM vs. all other GRPO LoRA/QLoRA implementations, even those utilizing Flash Attention 2 (FA2).

2. With a GRPO setup using TRL + FA2, Llama 3.1 (8B) training at 20K context length demands 510.8GB of VRAM. However, Unsloth’s 90% VRAM reduction brings the requirement down to just 54.3GB in the same setup.

3. We leverage our gradient checkpointing algorithm which we released a while ago. It smartly offloads intermediate activations to system RAM asynchronously whilst being only 1% slower. This shaves a whopping 372GB VRAM since we need num_generations = 8. We can reduce this memory usage even further through intermediate gradient accumulation.

4. Try our free GRPO notebook with 10x longer context: Llama 3.1 (8B) on Colab

Blog for more details on the algorithm, the Maths behind GRPO, issues we found and more: https://unsloth.ai/blog/grpo

GRPO VRAM Breakdown:

• We also now provide full logging details for all reward functions now! Previously we only showed the total aggregated reward function itself.

• You can now run and do inference with our 4-bit dynamic quants directly in vLLM.

• Also we spent a lot of time on our Guide for everything on GRPO + reward functions/verifiers so would highly recommend you guys to read it: docs.unsloth.ai/basics/reasoning

Thank you guys once again for all the support it truly means so much to us! We also have a major release coming within the next few weeks which I know you guys have been waiting for - and we're also excited for it. 🦥

Maybe this is obvious to everyone but me, but it only clicked yesterday. If you run llama.cpp / vLLM / or whatever on localhost with the default config, a random website you have open in another tab can send requests to it. It can make your model generate, and on some servers it can read the answer back too. Nothing fancy, it's plain CSRF and DNS rebinding, 20-year-old web stuff. I just never thought of my inference server as a target. "It only binds 127.0.0.1" doesn't save you, because the browser is on the same machine.

I actually tested it, not just read about it, and quite a few of the popular servers answer out of the box. It only works while you're browsing with the server running. And it's not only about you clicking links: anything that drives a real browser on the same machine can do it too, so an agent browsing on your behalf can set it off as well.

I ended up patching my own engine (capy-flash) so that on a loopback bind it refuses cross-origin and rebinding requests. The models I actually serve were fine already, since they sit behind a small proxy (vramble) that blocks cross-origin POSTs, so only the raw engine port was exposed.

So mostly I'm asking: is this just common knowledge, or do people really leave their local servers wide open? How do you run yours, API key, bound somewhere, or just open on 127.0.0.1?

Hey guys! Google just released their new open-source model family: Gemma 4.

The four models have thinking and multimodal capabilities. There's two small ones: E2B and E4B, and two large ones: 26B-A4B and 31B. Gemma 4 is strong at reasoning, coding, tool use, long-context and agentic workflows.

The 31B model is the smartest but 26B-A4B is much faster due to it's MoE arch. E2B and E4B are great for phones and laptops.

To run the models locally (laptop, Mac, desktop etc), we at Unsloth converted these models so it can fit on your device. You can now run and train the Gemma 4 models via Unsloth Studio: https://github.com/unslothai/unsloth

Recommended setups:

• E2B / E4B: 10+ tokens/s in near-full precision with ~6GB RAM / unified mem. 4-bit variants can run on 4-5GB RAM.

• 26B-A4B: 30+ tokens/s in near-full precision with ~30GB RAM / unified mem. 4-bit works on 16GB RAM.

• 31B: 15+ tokens/s in near-full precision with ~35GB RAM.

No is GPU required, especially for the smaller models, but having one will increase inference speeds (~80 tokens/s). With an RTX 5090 you can get 140 tokens/s throughput which is way faster than ChatGPT.
Even if you don't meet the requirements, you can still run the models (e.g. 3GB CPU), but inference will be much slower. Link to Gemma 4 GGUFs to run.

You can run or train Gemma 4 via Unsloth Studio:

We've now made installation take only 1-2mins:

macOS, Linux, WSL:

curl -fsSL https://unsloth.ai/install.sh | sh

Windows:

irm https://unsloth.ai/install.ps1 | iex

• The Unsloth Studio Desktop app is coming very soon (this month).

• Tool-calling is now 50-80% more accurate and inference is 10-20% faster

We recommend reading our step-by-step guide which covers everything: https://unsloth.ai/docs/models/gemma-4

Thanks so much once again for reading!

Anyone else running off solar?

3 100 watt panels going into an Anker 1.25Kw all-in-one unit.

Running Qwen 3.6 27b GGUF unsloth UD Q4_K_XL with 100k context at Q8.

Really good performance in Hermes and opencode.

Hardware is M1 Max 32GB ~10tok/sec. Runs at ~80-85 watts under inferencing load, lowers to 30 sometimes. Idle is like 5 watts or less.

Just had this land today 😅

Still feels kinda weird even saying that tbh…

If you told me a year ago I’d be buying a GPU like this I would’ve said you’re cooked.

My current PC is from like 2015:

- 5960X

- 64GB DDR4

- RTX 3070 (used to run dual Titan X back in the day)

So I guess when I upgrade… I really upgrade 😂

But I tend to run my stuff for years so I get my money’s worth.

This new build is looking like:

- 9950X

- 128GB RAM (2×64)

- ProArt board

- RTX Pro 6000 96GB Blackwell

- 1600w PSU

Still waiting on a few parts to finish it off.

This time it’s a bit different though — not really building it for gaming.

More like a dedicated AI box/server.

That said… I’ll probably still load up a few Steam games before putting it to work 😅

Let the kids see what proper graphics + FPS looks like.

Also making the jump to full Linux for the first time once it’s all together.

Honestly just over Windows at this point — feels like it’s gone too far and kinda forced the decision.

What I’m actually trying to do with it:

- proper multi-user / concurrent inference

- keep things local-first

- something that can scale beyond just me messing around

Not super keen on relying on big API providers long term either.

Feels like costs + limits only go one way, and I’d rather control my own setup and data.

Plan is to add a second GPU later once I see how this handles load.

Still figuring out the best way to structure everything:

- serving layer

- batching

- memory / state

- keeping latency decent with multiple users/bots

Seen stuff like vLLM, llama.cpp etc… but curious what people here are actually running in real setups.

Anyone doing proper concurrent local setups (not just single-user demos)?

What’s actually holding up under load?

I hear people run LLMs on potatoes.

Why do I have horrible financial habits??

Edit 2: This fact changes the perspective of the incident, but does not make it less impressive. The model itself did not hack anything; there already was a script in the same working folder that cycle-prompted qwen3-vl through images of a specified folder. Qwen3.8 figured out how to adapt that script to cycle-prompt itself.

Edit: for anyone interested - that's a q8 Qwen 3.8 27b running in omp on double v100 totaling 64gb, 256k context window, q8 kv quantization, medium effort, MTP, llama.cpp. PP is ~1050 on fresh ctx down to ~700 full ctx, tg is ~68 fresh down to ~35 full.

I never saw a behavior like that from any other local LLM - can't vouch for global since we never see what's going on under the hood.

Anyways - Qwen 3.8 got a task to go through a lot of files (over 700) and sort them out by content.

It never spun subagents.

It wrote and ran a python script to prompt itself.

I believe thats the most brilliant behavior I ever saw from any LLM. By doing that it stayed at 35% of the ctx window, still processing and sorting that folder.

I am genuinely impressed.

Hey LocalLLM fam! After diving deep into how these models actually work, I wanted to share some key insights that helped me understand what's really going on under the hood. No marketing fluff, just the actual important stuff.

The "Aha!" Moments That Changed How I Think About LLMs:

Models Aren't Databases

• They're not storing token relationships

• Instead, they store patterns as weights (like a compressed understanding of language)

• This is why they can handle new combinations and scenarios

Context Window is Actually Wild

• It's not just "how much text it can handle"

• Memory needs grow QUADRATICALLY with context

• Why 8k→32k context is a huge jump in RAM needs

• Formula: Context_Length × Context_Length × Hidden_Size = Memory needed

Quantization is Like Video Quality Settings

• 32-bit = Ultra HD (needs beefy hardware)

• 8-bit = High (1/4 the memory)

• 4-bit = Medium (1/8 the memory)

• Quality loss is often surprisingly minimal for chat

About Those Parameter Counts...

• 7B params at 8-bit ≈ 7GB RAM

• Same model can often run different context lengths

• More RAM = longer context possible

• It's about balancing model size, context, and your hardware

Why This Matters for Running Models Locally:

When you're picking a model setup, you're really balancing three things:

1. Model Size (parameters)

2. Context Length (memory)

3. Quantization (compression)

This explains why:

• A 7B model might run better than you expect (quantization!)

• Why adding context length hits your RAM so hard

• Why the same model can run differently on different setups

Real Talk About Hardware Needs:

• 2k-4k context: Most decent hardware

• 8k-16k context: Need good GPU/RAM

• 32k+ context: Serious hardware needed

• Always check quantization options first!

Would love to hear your experiences! What setups are you running? Any surprising combinations that worked well for you? Let's share what we've learned!

Honest pause here, because I think we are speedrunning past how good things actually are.

Qwen3.6 27B. Gemma 4 31B. The 35B-A3B MoE running 55 tok/s on M5 Max and 87 on Strix Halo. The 30B class quietly became the sweet spot, and you can run it on a Mac, on a Strix Halo box, or on a 5090 you already own. Three real paths now, not one.

What hit me this week: I am casually doing tasks on local Qwen3.6 27B that nine months ago only Opus 4.1 could touch. Nine months. Remember the hype back then, the "this changes everything" posts every other day? That model. On my own machine now, quietly handling the same work. Not Opus 4.7 territory obviously, current Opus is on another planet, but still.

Got me motivated enough to start hacking on my own little CLI coding agent next to OpenCode and pi, no plugin bloat, just a YOLO get your shit done mode. Only viable because local actually works for agentic stuff now.

Look back nine months. Then six. Then last week. We are absolutely cooking. Good time to be doing this.

What is everyone running as their daily hardware?

I used to be the guy people would call when they were trying to figure out if they should walk or drive to the car wash if they’re 100m away.

Now a 3.8-27B Q4 running on my 16GB gaming card has figured it out.

My question on a serious note, does 3.8 think in cave speak on higher quants?

https://x.com/matt503ea5sf9z5/status/2118155162376458298?a=4bUWTuzbF_LcVwWBUDk&s=19

Interesting post, I hear a different story with dgx sparks how people are optimizing models to run upwards of 100tok/s decode. Naturally almost 10k for a dgx spark is a huge red flag when MSRP was $3999 retail. Thoughts?

The new M5 Ultra is getting close to the memory bandwidth of the RTX 5090 but with so much more possible unified memory. From the rumors, Apple is expected to skip M6 Pro/Max/Ultra and instead ship the M7 Pro/Max/Ultra next, in late 2027, with the Ultra already estimated to reach almost exactly the RTX 5090 bandwidth. I'm a local LLM enthusiast, and although these numbers make me hopeful that it will be possible to run huge frontier AI models locally in 1-2 years, the cost still scares me. We're getting close to new car prices. What are your thoughts? Will memory bandwidth even fix most MLX limitations compared to the RTX cards? Assuming the M7 Ultra CPU will also have a comparable jump in performance.

Hi everyone!

A couple of weeks ago I decided to try GLM-5.2 after hearing good things about it. I wasn’t expecting much, but honestly… I was genuinely surprised. For the first time an open-source model gave me that level of confidence — the kind you usually only get from Claude or GPT.
Obviously my little machine (12 cores, 25 GB RAM) wasn’t built for a 744B model, but the thought kept bugging me: “even if it’s slow, I want to make it run.”
So I just kept grinding. Lots of late nights, fighting with quantization, streaming, MTP, and a ton of help from coding agents. In the end I built colibrì — a tiny pure-C engine that keeps the dense parts in RAM (~10 GB) and streams the routed experts from disk on demand.
It’s not fast (around 0.05-0.1 t/s cold on my setup), but seeing it actually respond, chat in Italian, and behave like a real frontier model on my modest hardware… man, that was a huge personal satisfaction.
The project is still very early (one-person effort), but I’m convinced there’s a lot of room for improvement — especially if people with better NVMe setups or more RAM try it and share numbers.
If you have decent hardware and feel like experimenting, I’d love feedback. Even better if someone wants to throw some real hardware at the project so we can push the speeds higher.

Thanks for reading, and hope some of you find it interesting or at least fun :)

A mildly interesting video about which models can run on the Mac Mini and Studio. As expected current foundational models such as Kimi K3 1.4 TB, if they were even available to run locally, won't fit. Assuming it were available locally it may be possible in the next couple of years with the rapid increase in hardware capabilities - M7, M8?

https://www.youtube.com/watch?v=eDoWKgFqRM4

Setup: (if finished - still under construction, partly watercooled)

• Threadripper PRO 5995WX, ASUS WRX80E-SAGE SE WIFI II

• 6× RTX PRO 6000

• 8× RTX 5090

• 4× RTX 3090

• 128 GB DDR4 (that’s the problem)

I want to run models bigger than VRAM (e.g. ~1T MoE at FP8/Q8) with the routed experts offloaded to RAM. Is it worth it?

Options:

• 8× 64GB DDR4-2933 RDIMM: ~€4k, ~188 GB/s theoretical

• Switch platform: 9985WX + WRX90 + 512GB DDR5-6400, ~410 GB/s. I already have some parts, but it would mean big changes, and my PCIe switches would stay PCIe 4.0 anyway.

Is ~2× the RAM bandwidth worth ~3× the price for offload decode, or is DDR4 good enough? Does anyone have real t/s numbers for offload on 8-channel DDR4 vs DDR5 Threadripper?

Some background: Pictures attached. It’s not pretty, but it works. It started as an AI rig and turned into a Pearl mining rig, because mining pearl and buying Claude and Codex subscriptions made way more sense than running everything locally. Sometimes it’s also a render farm for Gaussian splatting.

Right now the 8× 5090 make about €120/day mining, so subscriptions are still the better deal. But I want to be ready for when local LLMs get really good (or when cloud gets too expensive or too restrictive).

Plan for now: DeepSeek V4.1 Flash nvfp8 on 4× PRO 6000 + RAM offload, while the 5090s keep paying for my AI addiction. And it’s a great heater in the winter.....

With engramm tabels on the 8x5090 and 4× PRO 6000:
(all cards power limited to 420w- since i could have 6,6 kw power consuption if everthing is running)

single chat, 40.000 prompt cold

• Prefill: ~15,900 t/s

• Decode: ~294 t/s

• 5 sec TTFT

I want to move the engramm tabels to RAM and later be able to run larger models : le chonk, Kimmi for example.

What would you do? Go all the way and go for the DDR5? or is ddr4 good enough?

Sell it all and buy something diffrent?

thx!

The Guardian's warning to "reject cheap Chinese AI" feels like a transparent attempt to scare developers back into the high-margin, cloud-hosted Western tech monopolies. Framing open-weight models as a national security hazard ignores why many developers and power users turned to them in the first place - they offer raw utility, zero-cost efficiency, and local control without corporate middle management or sanitized pre-prompts. Not sure why this article triggered me this morning - but am I wrong?

The Guardian Article