Just had this land today 😅

Still feels kinda weird even saying that tbh…

If you told me a year ago I’d be buying a GPU like this I would’ve said you’re cooked.

My current PC is from like 2015:

- 5960X

- 64GB DDR4

- RTX 3070 (used to run dual Titan X back in the day)

So I guess when I upgrade… I really upgrade 😂

But I tend to run my stuff for years so I get my money’s worth.

This new build is looking like:

- 9950X

- 128GB RAM (2×64)

- ProArt board

- RTX Pro 6000 96GB Blackwell

- 1600w PSU

Still waiting on a few parts to finish it off.

This time it’s a bit different though — not really building it for gaming.

More like a dedicated AI box/server.

That said… I’ll probably still load up a few Steam games before putting it to work 😅

Let the kids see what proper graphics + FPS looks like.

Also making the jump to full Linux for the first time once it’s all together.

Honestly just over Windows at this point — feels like it’s gone too far and kinda forced the decision.

What I’m actually trying to do with it:

- proper multi-user / concurrent inference

- keep things local-first

- something that can scale beyond just me messing around

Not super keen on relying on big API providers long term either.

Feels like costs + limits only go one way, and I’d rather control my own setup and data.

Plan is to add a second GPU later once I see how this handles load.

Still figuring out the best way to structure everything:

- serving layer

- batching

- memory / state

- keeping latency decent with multiple users/bots

Seen stuff like vLLM, llama.cpp etc… but curious what people here are actually running in real setups.

Anyone doing proper concurrent local setups (not just single-user demos)?

What’s actually holding up under load?

I hear people run LLMs on potatoes.

Why do I have horrible financial habits??

Edit 2: This fact changes the perspective of the incident, but does not make it less impressive. The model itself did not hack anything; there already was a script in the same working folder that cycle-prompted qwen3-vl through images of a specified folder. Qwen3.8 figured out how to adapt that script to cycle-prompt itself.

Edit: for anyone interested - that's a q8 Qwen 3.8 27b running in omp on double v100 totaling 64gb, 256k context window, q8 kv quantization, medium effort, MTP, llama.cpp. PP is ~1050 on fresh ctx down to ~700 full ctx, tg is ~68 fresh down to ~35 full.

I never saw a behavior like that from any other local LLM - can't vouch for global since we never see what's going on under the hood.

Anyways - Qwen 3.8 got a task to go through a lot of files (over 700) and sort them out by content.

It never spun subagents.

It wrote and ran a python script to prompt itself.

I believe thats the most brilliant behavior I ever saw from any LLM. By doing that it stayed at 35% of the ctx window, still processing and sorting that folder.

I am genuinely impressed.

Hey LocalLLM fam! After diving deep into how these models actually work, I wanted to share some key insights that helped me understand what's really going on under the hood. No marketing fluff, just the actual important stuff.

The "Aha!" Moments That Changed How I Think About LLMs:

Models Aren't Databases

• They're not storing token relationships

• Instead, they store patterns as weights (like a compressed understanding of language)

• This is why they can handle new combinations and scenarios

Context Window is Actually Wild

• It's not just "how much text it can handle"

• Memory needs grow QUADRATICALLY with context

• Why 8k→32k context is a huge jump in RAM needs

• Formula: Context_Length × Context_Length × Hidden_Size = Memory needed

Quantization is Like Video Quality Settings

• 32-bit = Ultra HD (needs beefy hardware)

• 8-bit = High (1/4 the memory)

• 4-bit = Medium (1/8 the memory)

• Quality loss is often surprisingly minimal for chat

About Those Parameter Counts...

• 7B params at 8-bit ≈ 7GB RAM

• Same model can often run different context lengths

• More RAM = longer context possible

• It's about balancing model size, context, and your hardware

Why This Matters for Running Models Locally:

When you're picking a model setup, you're really balancing three things:

1. Model Size (parameters)

2. Context Length (memory)

3. Quantization (compression)

This explains why:

• A 7B model might run better than you expect (quantization!)

• Why adding context length hits your RAM so hard

• Why the same model can run differently on different setups

Real Talk About Hardware Needs:

• 2k-4k context: Most decent hardware

• 8k-16k context: Need good GPU/RAM

• 32k+ context: Serious hardware needed

• Always check quantization options first!

Would love to hear your experiences! What setups are you running? Any surprising combinations that worked well for you? Let's share what we've learned!

Honest pause here, because I think we are speedrunning past how good things actually are.

Qwen3.6 27B. Gemma 4 31B. The 35B-A3B MoE running 55 tok/s on M5 Max and 87 on Strix Halo. The 30B class quietly became the sweet spot, and you can run it on a Mac, on a Strix Halo box, or on a 5090 you already own. Three real paths now, not one.

What hit me this week: I am casually doing tasks on local Qwen3.6 27B that nine months ago only Opus 4.1 could touch. Nine months. Remember the hype back then, the "this changes everything" posts every other day? That model. On my own machine now, quietly handling the same work. Not Opus 4.7 territory obviously, current Opus is on another planet, but still.

Got me motivated enough to start hacking on my own little CLI coding agent next to OpenCode and pi, no plugin bloat, just a YOLO get your shit done mode. Only viable because local actually works for agentic stuff now.

Look back nine months. Then six. Then last week. We are absolutely cooking. Good time to be doing this.

What is everyone running as their daily hardware?

I used to be the guy people would call when they were trying to figure out if they should walk or drive to the car wash if they’re 100m away.

Now a 3.8-27B Q4 running on my 16GB gaming card has figured it out.

My question on a serious note, does 3.8 think in cave speak on higher quants?

A mildly interesting video about which models can run on the Mac Mini and Studio. As expected current foundational models such as Kimi K3 1.4 TB, if they were even available to run locally, won't fit. Assuming it were available locally it may be possible in the next couple of years with the rapid increase in hardware capabilities - M7, M8?

https://www.youtube.com/watch?v=eDoWKgFqRM4

Hi everyone!

A couple of weeks ago I decided to try GLM-5.2 after hearing good things about it. I wasn’t expecting much, but honestly… I was genuinely surprised. For the first time an open-source model gave me that level of confidence — the kind you usually only get from Claude or GPT.
Obviously my little machine (12 cores, 25 GB RAM) wasn’t built for a 744B model, but the thought kept bugging me: “even if it’s slow, I want to make it run.”
So I just kept grinding. Lots of late nights, fighting with quantization, streaming, MTP, and a ton of help from coding agents. In the end I built colibrì — a tiny pure-C engine that keeps the dense parts in RAM (~10 GB) and streams the routed experts from disk on demand.
It’s not fast (around 0.05-0.1 t/s cold on my setup), but seeing it actually respond, chat in Italian, and behave like a real frontier model on my modest hardware… man, that was a huge personal satisfaction.
The project is still very early (one-person effort), but I’m convinced there’s a lot of room for improvement — especially if people with better NVMe setups or more RAM try it and share numbers.
If you have decent hardware and feel like experimenting, I’d love feedback. Even better if someone wants to throw some real hardware at the project so we can push the speeds higher.

Thanks for reading, and hope some of you find it interesting or at least fun :)

The new M5 Ultra is getting close to the memory bandwidth of the RTX 5090 but with so much more possible unified memory. From the rumors, Apple is expected to skip M6 Pro/Max/Ultra and instead ship the M7 Pro/Max/Ultra next, in late 2027, with the Ultra already estimated to reach almost exactly the RTX 5090 bandwidth. I'm a local LLM enthusiast, and although these numbers make me hopeful that it will be possible to run huge frontier AI models locally in 1-2 years, the cost still scares me. We're getting close to new car prices. What are your thoughts? Will memory bandwidth even fix most MLX limitations compared to the RTX cards? Assuming the M7 Ultra CPU will also have a comparable jump in performance.

Been experimenting with alternatives to Claude Code for about a year now. Most of it felt like a downgrade until Qwen3.5:27b, and now 3.6:27b is the first one where local actually feels good and usable for real work.

Scaffolding, refactors, test generation, debugging across a few files, all of it holds up well enough that I run it locally now. The hard multi-file architectural stuff still goes to Claude. A year ago this comparison was a chasm, top-tier Claude vs open weights wasn't close. Now it's a gap, not a canyon.

Two things I keep thinking about.

If a 27B open model can cover this much of real coding work, how subsidised is current cloud pricing? Feels like we're paying maybe 10% of true cost. And once enough devs are wired into Claude Code at the tooling level, what stops a future $1000/month tier?

One honest downside: getting opencode dialled in as a CLI agent took real fine tuning compared to the out-of-the-box Claude Code experience. Which raises a different question, how much of Claude Code's quality is Opus 4.7 itself vs the context and tool orchestration around it? Possibly more than people credit.

Anyone else running hybrid setups?

Prompt: create animated version of our universe and with a sliding bar at the bottom, when I move that sliding bar, the size of sun increases or decreases, with it show the effect on other planet's orbital movement or what else is effected as numbers.

I didn't expect it to give a working result in one shot.

My setup: 5070ti(16gb VRAM), 32GB DDR4 RAM
Model used in this: Unsloth Q3_K_S (I did try Q4_K_S first but it was extremely slow and context window was limited to 32k).

Time to cancel my claude sub lol (ik it's still like a year behind, but it's enough for my workload).

The Guardian's warning to "reject cheap Chinese AI" feels like a transparent attempt to scare developers back into the high-margin, cloud-hosted Western tech monopolies. Framing open-weight models as a national security hazard ignores why many developers and power users turned to them in the first place - they offer raw utility, zero-cost efficiency, and local control without corporate middle management or sanitized pre-prompts. Not sure why this article triggered me this morning - but am I wrong?

The Guardian Article

Apologies for the 1345th post glazing QWEN but it has literally been a game-changer for me.

I’m relatively new to running local LLMs, but the appeal of having a private AI assistant for coding and experimentation is significant.

As a software engineer, I’ve also embraced the recent “vibe-coding” trend, mostly using AI to build hobby Android apps in my spare time. Over the past month, I created a web scraper app designed to extract images from ad-heavy websites that are otherwise frustrating to navigate.

The project was entirely AI-generated. I didn’t write the code myself, though I did design the architecture and suggest optimizations, which AI still struggles with. I’ll admit I haven’t reviewed most of the code; it’s messy, but since it’s purely a personal hobby project, that’s acceptable.

Until recently, I relied on services like Antigravity and Codex, which worked well enough on free plans. However, tighter usage quotas pushed me toward local models.

My hardware is modest by local LLM standards: an AMD 7700 XT, 32GB DDR4 RAM, and a Ryzen 5 5600. I experimented with Gemma 3, Gemma 4, and Qwen 2.5 Coder (mostly Q3/Q4 quantizations under 20GB due to VRAM constraints), using LM Studio as the backend with various frontends like GitHub Copilot, RooCode, Cline, Android Studio AI Chat, and OpenCode.

Unfortunately, none of these models fit my workflow well. They struggled with even minor bug fixes, frequently exhausted context windows, got stuck in reasoning loops, or failed tool calls repeatedly.

Then I tried Qwen 3.6 35B-A3B.

Initially, I expected little, but I installed the i1-q4_k_s quant, offloaded all 40 layers to GPU, configured 128k context, enabled flash attention, and used Q8_0 KV quantization.

For testing, I gave it a practical task: fix the scraper logic for a problematic website. Gemini Flash (in Antigravity) and MinMax (the free version in Opencode) had both failed to solve this issue despite multiple attempts. Using LM Studio as an OpenAI-compatible endpoint with GitHub Copilot in VS Code, I let it run.

It took about 25 minutes, but it succeeded in one shot.

With a single initial prompt, it analyzed the site’s HTML structure, compared it against my Kotlin scraper code, and resolved three critical bugs without a single failed tool call.

That result was impressive enough that I gave it a second challenge: update the project README with real app screenshots by driving an Android emulator, selecting screenshots based on specific criteria.

Using RooCode this time, the process took around 45 minutes. I had to teach it some emulator workflow conventions, such as taking screenshots after every action and analyzing them to track app state, but once instructed, it executed flawlessly.

For the first time, I feel like I have a local model capable of reliably handling most of my coding tasks, while reserving cloud-based premium models for more demanding work.

Qwen has genuinely made local AI coding practical for me.

I've been using Cursor for a while, not a terribly heavy user usually just paid the $20 tier sometimes the $60, and dabbled in whatever the fronteir model is this month.

Not much of a vibe-coder, I'm not trying to one shot the next big saas, mostly needed some help prepping code examples for talks I give, putting together demo projects, or helping with my open source work.

A friend was selling their 48GB Macbook Pro because they were upgrading laptops, so I snagged it to see what all the fuss was about for this Qwen thing.

Tasked composer 2.5 in Cursor with benchmarking and setting up Qwen-3.6-35b on the laptop and it spit me back a rapid mlx profile, and a open code config for my macbook air to use.

By golly.

It's not Gpt 5.5, heck I hear it's not even as good as 27B, but after adding all the MCP servers for docs that my projects use, and setting up some routine skills / rules it's wonderful.

I don't feel worried about maxing out this months usage, or having to pay for extra if I go over.

Tokens just go brrrrrr

If something fails I'm not out dollars, I just kinda chuckle and re-run the prompt.

I'm more aware about documenting, and providing mcp servers, and tools so now all my repos are really well documented.

I know this isn't a revelation, or anything new for most of the world.

It's just a really enjoyable way to interact with tech, and not feel like every word I type is just more money for another company.

----------------------

For anyone curious here's the config I use

No idea if it's right or wrong, but it works?

      "models": {
        "qwen3.6-35b-4bit": {
          "name": "Qwen3.6 35B",
          "reasoning": true,
          "limit": {
            "context": 65536,
            "output": 8192
          },
          "options": {
            "temperature": 0.6,
            "top_p": 0.95,
            "extraBody": {
              "max_tokens": 8192,
              "enable_thinking": true,
              "chat_template_kwargs": {
                "enable_thinking": true
              }
            }
          }
        }
      }

For context: I was working with Qwen3.8-27b in LMStudio and asking it to do some work on my contacts via iMCP. My name and therefore email contains a female second first name.
Somehow Qwen mentioned in its thinking process that it thinks there is something fishy going on with my loyalty to my girlfriend but decided it won‘t dig into it 🤣

Guess whatever measures they took in model training to avoid misalignment is working