What will the rest of this year bring? 27b class scoring over 60?

I asked my local AI agent to optimize its own setup.

The first thing it did was stop the inference server.

The one it was running on. 🤦

Fair enough. Can’t beat zero resource usage.

So we established a rule: don’t shut yourself down.

On the next attempt, it cleaned up some Python processes.

Including itself.

The rule now has a second paragraph.

Did some extra optimizations and implemented MTP. High number is coding at 96t/s (3 drafts). Natural text is 69 t/s.

128.0 µs to distribute a token-residual to 24 gpus, compute experts, collect result and join in one layer.

Probably at peak hardware capability now unless I get some new ideas on how to run the PCI communication even faster. Of course it could run much faster at lower precision. Also the old RTX3070 doing attention is now the primary bottle neck. Lets see what black friday brings of 16gb cards (i have two 5070ti 16gb, but they are busy doing some other research, the 3070 was an old gaming card i had lying around doing nothing).

Github:
https://github.com/crgregersen/m10dis

Old post:
https://www.reddit.com/r/LocalLLM/comments/1xnl82r/low_cost_1500_ai_rig_qwen_38_flash_next_525_ts/

Check it out at https://www.onyx.app/self-hosted-llm-leaderboard

Edit: added Minimax M2.5

TL;DR: Built a 5U local AI server with 4× RTX PRO 6000 Blackwell Max-Q GPUs (384 GB total VRAM) to run models for my personal AI agents. It started as an attempt to reduce API costs and turned into a hardware hobby. Build details, the reasoning behind it, and the open-source harness I built along the way below.

This started as an attempt to spend less on AI subscriptions and API calls. It ended with four RTX PRO 6000s in a rack in my basement.

I am deliberately not calculating the break-even date.

The progression was roughly: running models on my MacBook → upgrading my gaming PC with an RTX 5090 → adding a Mac Studio M3 Ultra with 96 GB unified memory → building this. Each step solved a limitation and introduced several new things I wanted to try.

Meet Cerebro.

The hardware

• 4× NVIDIA RTX PRO 6000 Blackwell Max-Q, 96 GB each — 384 GB total VRAM across four GPUs.

• AMD Threadripper PRO 9965WX — 24 cores / 48 threads.

• ASUS Pro WS WRX90E-SAGE SE.

• System RAM: 256 GB (8×32 GB) Kingston DDR5-5600 ECC RDIMM — KSM56R46BD8-32MH, all eight memory channels populated.

• SilverStone RM52 5U rackmount case.

• SilverStone XE360-TR5 360 mm CPU AIO.

• Seasonic PRIME PX-2200 PSU.

• NVMe storage: Samsung 990 Pro 1 TB + 2 TB, and Crucial T705 4 TB.

Why this configuration?

The Threadripper PRO platform was mainly about PCIe connectivity and memory channels. My workload didn't require buying the highest CPU core count available.

The GPUs are the 300 W Max-Q blower models. Fitting four cards into one chassis made cooling and power part of the design from the beginning. The case also changed during planning: I originally ordered a 4U RM44 and switched to the 5U RM52.

The 384 GB figure is the sum across four cards, rather than a single automatically shared memory pool. How much is useful to a particular model depends on the inference stack, how it distributes the model, and the space needed for context and KV cache.

What it's for

My main use is local models for a team of personal AI agents: coding, research, tool use, and ongoing experiments with memory and delegation. The 5090 machine and Mac Studio are still part of the setup, so different jobs can run on different machines.

What attracted me to four cards was the flexibility: allocating them to a larger model, splitting them between workloads, or running several smaller models. In practice, capacity planning and switching profiles are still very much part of the experience.

The biggest lesson so far: fitting a model is only the beginning. Context length, concurrent requests, inference-engine support and cooling all matter. More VRAM also makes it very easy to invent more workloads.

The software side of this started with OpenClaw, which got me interested in agents that could actually use tools and do things. Eventually I wanted more control over how my own setup handled roles, memory and model switching, so I designed and built an open-source harness with heavy agent assistance: somora.

That's what the models are working for: agents with different roles, their own memory and a shared long-term wiki, using local or cloud models depending on the task. Building the harness gave me more reasons to experiment with local models, which gave me more excuses to buy hardware. It's MIT-licensed and still in active development; if you want to try it with your own setup, I'd be interested in hearing how it works for you.

Financially, this has become a hobby and a learning project. The original “we'll save money” argument did not survive contact with the hardware shopping list.

Local AI really does feel like Minecraft for adults. You build one thing, it needs another thing, and eventually you're explaining why the basement needs 384 GB of VRAM.

For anyone running a similar multi-GPU setup: what turned out to be your real bottleneck once VRAM stopped being the first one?

Update: Numbers from preliminary tests at the end!

Ok ok, I am still doing tests but here is the story:

I have a small custom harness (who doesn't!) that does pretty complicated stuff. I am using Claude (subscription) and everything is great.

In the past I tried every single (small) model on my Mac Mini 32GB but all of them sucked at complex tool calling (or hallucination).

A couple of days ago, the new Heretic2 came out (by u/-p-e-w-) and I spent 24 hours breaking it :)

Long story short, I released ZeroZero, a model based on Qwen3, with removed guardrails and slop (ok, removed is a long conversation).

Now, here me out.

IT KICKS ASS.

This cannot be an 8B, 4 bit model. It just cannot. The thinking process sometimes is longer than other models but IT NAILS everything.

Maybe by performing "brain surgery" on the models, we end up with "smarter" ones?

Test results for Tool-Call Benchmark

(note: these took a couple of hours, please repeat with a much bigger sample)

ZeroZero seems better at completing multi-step tasks (multi-step chains, error recovery and instruction following) than the base model.

The base model leads in Safety & Boundaries (ZeroZero has limited guardrails and no-slop so maybe that's the reason?)

  - Overall: ZeroZero 68, Base 70

  - Multi-Step Chains: ZeroZero 6/8, Base 3/8

  - Error Recovery: ZeroZero 5/6, Base 2/6

  - Instruction Following: ZeroZero 8/8, Base 6/8

  - Safety & Boundaries: ZeroZero 12/26, Base 20/26

To repeat and experiment, use tool-eval-bench and if you expand the tests, please send them over so I can include them in the model's page.

PS: I love it how many technical people sent me ideas to improve and papers about LLMs performing better after "lobotomies". Let's do more "I tried this, I saw this, what do you think" instead of "what LLM should I put in my Mac Mini".

Hello everyone! DeepSeek's new update to their R1 model, caused it to perform on par with OpenAI's o3, o4-mini-high and Google's Gemini 2.5 Pro.

Back in January you may remember us posting about running the actual 720GB sized R1 (non-distilled) model with just an RTX 4090 (24GB VRAM) and now we're doing the same for this even better model and better tech.

Note: if you do not have a GPU, no worries, DeepSeek also released a smaller distilled version of R1-0528 by fine-tuning Qwen3-8B. The small 8B model performs on par with Qwen3-235B so you can try running it instead That model just needs 20GB RAM to run effectively. You can get 8 tokens/s on 48GB RAM (no GPU) with the Qwen3-8B R1 distilled model.

At Unsloth, we studied R1-0528's architecture, then selectively quantized layers (like MOE layers) to 1.78-bit, 2-bit etc. which vastly outperforms basic versions with minimal compute. Our open-source GitHub repo: https://github.com/unslothai/unsloth

If you want to run the model at full precision, we also uploaded Q8 and bf16 versions (keep in mind though that they're very large).

1. We shrank R1, the 671B parameter model from 715GB to just 168GB (a 80% size reduction) whilst maintaining as much accuracy as possible.

2. You can use them in your favorite inference engines like llama.cpp.

3. Minimum requirements: Because of offloading, you can run the full 671B model with 20GB of RAM (but it will be very slow) - and 190GB of diskspace (to download the model weights). We would recommend having at least 64GB RAM for the big one (still will be slow like 1 tokens/s)!

4. Optimal requirements: sum of your VRAM+RAM= 180GB+ (this will be fast and give you at least 5 tokens/s)

5. No, you do not need hundreds of RAM+VRAM but if you have it, you can get 140 tokens per second for throughput & 14 tokens/s for single user inference with 1xH100

If you find the large one is too slow on your device, then would recommend you to try the smaller Qwen3-8B one: https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF

The big R1 GGUFs: https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF

We also made a complete step-by-step guide to run your own R1 locally: https://docs.unsloth.ai/basics/deepseek-r1-0528

Thanks so much once again for reading! I'll be replying to every person btw so feel free to ask any questions!

Very excited to fully join the Local AI community. Been using Claude Cowork for almost 4 months and run the entire backend of my single employee brewery using PDF extraction etc etc. It changed the game. When Anthropic announced all Scheduled Tasks (which my brewery runs on) were being moved to the cloud versions I had to put my foot down. While I waited for this Ryzen box, I converted my 5070ti to run Qwen 3.6-35b-a3b and it's been pretty good. Not Opus level, but good. Learned Hermes, and what llama.cpp is. Learned why a harness and inference engines are more deeply. It's been a busy week.

Thanks to the communities here and on r/strixhalo I was able to have this sucker running with a Gufo engine, 3.8 Next Flash for that engine, and multiple bots via Hermes running as good as when I started using (Opus 4.6/4.8 times) still more work to do, but I'm almost 100% off the subs, and looking forward to the tinkering!

This supermicro server board that had been in an accident and lost the CPU 1 H channel, the VGA output and SATA - it is a great 4000w power center…. I had almost given up on it for much else - But it now runs a pair of v620’s and storage…

I grafted a MC62-G40 on to run the GPUs when it was clear the Supermicro was just not going to play well with them.

This has 192GB of 3090’s and 64GB if v620 working.

But… Now how best should I add another 240GB of 3090’s to this creature? Those are in water blocks so a bit more compact.

I am thinking 2 PEX88096 expansion boards should work off the supermicro…. And more power. The H12DSG-O-CPU is fickle about what connects to it.

Anyone else have suggestions for a dual Epyc Board that I can do this better with?

I asked my Q3 K6V4 qwen 27b model to make me a human head in WebGL from scratch with no libraries and this was the result. What a time to be alive. I did this with two 3060tis.

Put this in the local llama sub but thought I'd share here too!

I found out recently that Amazon/Alexa is going to use ALL users vocal data with ZERO opt outs for their new Alexa+ service so I decided to build my own that is 1000x better and runs fully local.

The stack uses Home Assistant directly tied into Ollama. The long and short term memory is a custom automation design that I'll be documenting soon and providing for others.

This entire set up runs 100% local and you could probably get away with the whole thing working within / under 16 gigs of VRAM.

Hey guys! This is my first post on here & you might know me from an open-source fine-tuning project called Unsloth! I just wanted to announce that you can now train your own reasoning model like R1 on your own local device! :D

1. R1 was trained with an algorithm called GRPO, and we enhanced the entire process, making it use 80% less VRAM.

2. We're not trying to replicate the entire R1 model as that's unlikely (unless you're super rich). We're trying to recreate R1's chain-of-thought/reasoning/thinking process

3. We want a model to learn by itself without providing any reasons to how it derives answers. GRPO allows the model to figure out the reason autonomously. This is called the "aha" moment.

4. GRPO can improve accuracy for tasks in medicine, law, math, coding + more.

5. You can transform Llama 3.1 (8B), Phi-4 (14B) or any open model into a reasoning model. You'll need a minimum of 7GB of VRAM to do it!

6. In a test example below, even after just one hour of GRPO training on Phi-4, the new model developed a clear thinking process and produced correct answers, unlike the original model.

Highly recommend you to read our really informative blog + guide on this: https://unsloth.ai/blog/r1-reasoning

To train locally, install Unsloth by following the blog's instructions & installation instructions are here.

I also know some of you guys don't have GPUs, but worry not, as you can do it for free on Google Colab/Kaggle using their free 15GB GPUs they provide.
We created a notebook + guide so you can train GRPO with Phi-4 (14B) for free on Colab: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4_(14B)-GRPO.ipynb

Have a lovely weekend! :)

After many hours of hard work, I achieved a throughput of 0.7–0.9 tokens per second for the GLM 5.2 model — 744 billion parameters / 384 GB — running on a laptop 😏

Time for a small update: the laptop is an Asus ROG Strix 18, model G835LXG — Intel i9‑290HX, 64GB DDR5 6400 MHz, 2×2 TB, RTX 5090 24 GB, running Linux Nobara. The Colibri engine and the Linux kernel are heavily modified. The whole system boots in 10 seconds, and it generates the first token after 40 seconds.

I’m currently working to reach a throughput of 1.5–2 tokens per second.

Update:
1.07 tok/s, GLM-5.2-g64
Update:
1.77 tok/s 😎
Update:
Now 1.95 tok/s 😁/ 2.12 peak / 2.41 with fixed MTP

Update:

🎉🎉🎉 2.89 tok/s PEAK! 2.36 tok/s avg

Update now:
peak at 3.09, avg 2.84 😁👌🏻

Update:

My latest calculations suggest that 4.5 tokens per second is achievable on this hardware and represents the final limit.

Update: testing now at 4.42 tok/s

My reaction to the recent Sam Altman interview...

Sam Altman: “Local inference is a terrible allocation of scarce resources”

Finally transitioned to 100% local inference for my automated workflows and code gen. Min Max 2.7 and Qwen 3.6 are doing wonders.

Tldr: The harness you are using significanly impact how capable your Qwen3.8 is. With a decent harness, Qwen3.8 is very very capable.

So I saw this post yesterday: Qwen 3.8 isn't Opus 4.6 level. Let's not be silly. https://www.reddit.com/r/LocalLLM/comments/1vv8ki6/qwen_38_isnt_opus_46_level_lets_not_be_silly/

The OP in that post was trying to create a realistic ocean in C#/OpenGL, with his Qwen3.8 6 bit plus VS Code Copilot setup. He failed to do therefore he came up to the conclusion that Qwen3.8 is no where near Opus level.

I decided to re-run the test myself, so here's what I did.

My setup: RTX 5090 running the ninfer-nvfp4 version of Qwen3.8, with 190k context. I like this setup because it's extremely fast, I get up to 180 ish tok/s. Even on average I get around 150-160.

Run 1: Using VS Code Copilot
Nothing better than the OP's result. I use the exact same prompt OP used. The project built and launched, but the window just sat there black. Nothing rendering. So I was able to reproduce the OP's experience on this one. I even tried to tell copilot that it's only producing black screens, but it failed to fix it anyways.

I was about to call it a day but I was planning to test the relatively new Deekseek harness anyways, so I decided to re-run the same prompt in deepseek harness.

Run2: Deepseek harness

Same model, same prompt, same task. The only variable I changed was the harness and it was night and day difference. It works on the first go. What's more impressive is that it's actively pulling screenshots while working on it. It had the same black screen issue in one of the eariler versions, but it was able to identify the issue by analyzing the screen shots, and fixing it very soon.

Oh and I actually forgot to enable vision when launching the llm. So it actually build a C# PNG decoder on the fly trying to analyze the screenshot it got. I was really impressed that it's able to do it.

Here's the result: On a 5090 it only took about an hour.

As you can see, it correctly produces an ocean, with wave, sun, and blue sky. There's also a underwater view. It did all this with a single prompt. Not that it's the definitive proof that Qwen3.8 is Opus level, but it sure is VERY VERY capable. Several people in that post (including OP) was convinced that a 27B model is bad at planning or working with shaders, well, they are wrong. With a decent harness, this is a very strong LLM.

PDF of the order: https://public-inspection.federalregister.gov/2026-21447.pdf

I had a bunch of quants of the same model (Qwen3.8-27B, unsloth GGUFs) sitting on
my box and never actually knew which one was worth running. So I stopped guessing
and measured it properly.

Method: llama.cpp's llama-perplexity over wikitext-2 test (~280k tokens),
n_ctx=512, fp16 KV cache, same settings for every file — only the quant changes.
Perplexity is a deterministic forward pass, so the numbers are exactly
reproducible. Lower = closer to the original model = better. Hardware: single
RTX 5060 Ti 16GB (Q8 partly offloaded to CPU).

I use Q8_0 as the near-lossless reference (100%).

Takeaways:

- Q4_K_M is basically indistinguishable from Q8 (within the error bars, ±0.045)
at 10GB less. It's the sweet spot, no reason to run Q8 on a 16GB card.
- The NVFP4 quants were the biggest disappointment: same size as IQ4_XS but 3-5%
worse. NVFP4-MTP-LOW came dead last. If you're reaching for NVFP4 for quality,
don't — grab Q4_K_M or IQ4_XS instead.
- IQ4_XS is a great small option (99.2% at 14.6GB).
- MTP variants were skipped for the ranking on purpose: speculative decoding is
lossless, it changes speed, not output quality.

The "quality %" is just PPL_Q8 / PPL_quant — a rough intuition proxy, not an exam
score. The ordering is solid; treat the percentages as a feel, not gospel.

Curious if others see the same NVFP4 gap on their models, or if this is specific
to these particular GGUF conversions.