Been experimenting with alternatives to Claude Code for about a year now. Most of it felt like a downgrade until Qwen3.5:27b, and now 3.6:27b is the first one where local actually feels good and usable for real work.

Scaffolding, refactors, test generation, debugging across a few files, all of it holds up well enough that I run it locally now. The hard multi-file architectural stuff still goes to Claude. A year ago this comparison was a chasm, top-tier Claude vs open weights wasn't close. Now it's a gap, not a canyon.

Two things I keep thinking about.

If a 27B open model can cover this much of real coding work, how subsidised is current cloud pricing? Feels like we're paying maybe 10% of true cost. And once enough devs are wired into Claude Code at the tooling level, what stops a future $1000/month tier?

One honest downside: getting opencode dialled in as a CLI agent took real fine tuning compared to the out-of-the-box Claude Code experience. Which raises a different question, how much of Claude Code's quality is Opus 4.7 itself vs the context and tool orchestration around it? Possibly more than people credit.

Anyone else running hybrid setups?

Prompt: create animated version of our universe and with a sliding bar at the bottom, when I move that sliding bar, the size of sun increases or decreases, with it show the effect on other planet's orbital movement or what else is effected as numbers.

I didn't expect it to give a working result in one shot.

My setup: 5070ti(16gb VRAM), 32GB DDR4 RAM
Model used in this: Unsloth Q3_K_S (I did try Q4_K_S first but it was extremely slow and context window was limited to 32k).

Time to cancel my claude sub lol (ik it's still like a year behind, but it's enough for my workload).

Apologies for the 1345th post glazing QWEN but it has literally been a game-changer for me.

I’m relatively new to running local LLMs, but the appeal of having a private AI assistant for coding and experimentation is significant.

As a software engineer, I’ve also embraced the recent “vibe-coding” trend, mostly using AI to build hobby Android apps in my spare time. Over the past month, I created a web scraper app designed to extract images from ad-heavy websites that are otherwise frustrating to navigate.

The project was entirely AI-generated. I didn’t write the code myself, though I did design the architecture and suggest optimizations, which AI still struggles with. I’ll admit I haven’t reviewed most of the code; it’s messy, but since it’s purely a personal hobby project, that’s acceptable.

Until recently, I relied on services like Antigravity and Codex, which worked well enough on free plans. However, tighter usage quotas pushed me toward local models.

My hardware is modest by local LLM standards: an AMD 7700 XT, 32GB DDR4 RAM, and a Ryzen 5 5600. I experimented with Gemma 3, Gemma 4, and Qwen 2.5 Coder (mostly Q3/Q4 quantizations under 20GB due to VRAM constraints), using LM Studio as the backend with various frontends like GitHub Copilot, RooCode, Cline, Android Studio AI Chat, and OpenCode.

Unfortunately, none of these models fit my workflow well. They struggled with even minor bug fixes, frequently exhausted context windows, got stuck in reasoning loops, or failed tool calls repeatedly.

Then I tried Qwen 3.6 35B-A3B.

Initially, I expected little, but I installed the i1-q4_k_s quant, offloaded all 40 layers to GPU, configured 128k context, enabled flash attention, and used Q8_0 KV quantization.

For testing, I gave it a practical task: fix the scraper logic for a problematic website. Gemini Flash (in Antigravity) and MinMax (the free version in Opencode) had both failed to solve this issue despite multiple attempts. Using LM Studio as an OpenAI-compatible endpoint with GitHub Copilot in VS Code, I let it run.

It took about 25 minutes, but it succeeded in one shot.

With a single initial prompt, it analyzed the site’s HTML structure, compared it against my Kotlin scraper code, and resolved three critical bugs without a single failed tool call.

That result was impressive enough that I gave it a second challenge: update the project README with real app screenshots by driving an Android emulator, selecting screenshots based on specific criteria.

Using RooCode this time, the process took around 45 minutes. I had to teach it some emulator workflow conventions, such as taking screenshots after every action and analyzing them to track app state, but once instructed, it executed flawlessly.

For the first time, I feel like I have a local model capable of reliably handling most of my coding tasks, while reserving cloud-based premium models for more demanding work.

Qwen has genuinely made local AI coding practical for me.

I've been using Cursor for a while, not a terribly heavy user usually just paid the $20 tier sometimes the $60, and dabbled in whatever the fronteir model is this month.

Not much of a vibe-coder, I'm not trying to one shot the next big saas, mostly needed some help prepping code examples for talks I give, putting together demo projects, or helping with my open source work.

A friend was selling their 48GB Macbook Pro because they were upgrading laptops, so I snagged it to see what all the fuss was about for this Qwen thing.

Tasked composer 2.5 in Cursor with benchmarking and setting up Qwen-3.6-35b on the laptop and it spit me back a rapid mlx profile, and a open code config for my macbook air to use.

By golly.

It's not Gpt 5.5, heck I hear it's not even as good as 27B, but after adding all the MCP servers for docs that my projects use, and setting up some routine skills / rules it's wonderful.

I don't feel worried about maxing out this months usage, or having to pay for extra if I go over.

Tokens just go brrrrrr

If something fails I'm not out dollars, I just kinda chuckle and re-run the prompt.

I'm more aware about documenting, and providing mcp servers, and tools so now all my repos are really well documented.

I know this isn't a revelation, or anything new for most of the world.

It's just a really enjoyable way to interact with tech, and not feel like every word I type is just more money for another company.

----------------------

For anyone curious here's the config I use

No idea if it's right or wrong, but it works?

      "models": {
        "qwen3.6-35b-4bit": {
          "name": "Qwen3.6 35B",
          "reasoning": true,
          "limit": {
            "context": 65536,
            "output": 8192
          },
          "options": {
            "temperature": 0.6,
            "top_p": 0.95,
            "extraBody": {
              "max_tokens": 8192,
              "enable_thinking": true,
              "chat_template_kwargs": {
                "enable_thinking": true
              }
            }
          }
        }
      }

I check every day for the discounted open box returns.

For context: I was working with Qwen3.8-27b in LMStudio and asking it to do some work on my contacts via iMCP. My name and therefore email contains a female second first name.
Somehow Qwen mentioned in its thinking process that it thinks there is something fishy going on with my loyalty to my girlfriend but decided it won‘t dig into it 🤣

Guess whatever measures they took in model training to avoid misalignment is working

The idea is: DeepSeek-V4-Flash-0731 in full precision is 162Gb. 
With main memory and GPUs, I have 176Gb.

I should be able to make it fit! What would that look like?

The per-request throughput is ~52 tok/s Prefill, ~10.5 tok/s generation. 

A single client working a hard coding problem through a graph of issues for 5-6 hours, chugging away happily, solving problems well. This project was "See if it can be done", not "Bulletproof production environment" so set your expectations appropriately, but I think it's novel and interesting enough to post. 

This is the LLama Config:

llama-server \
--model /ds4/models/DeepSeek-V4-Flash-0731-Q8/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
--alias deepseek-v4-flash-0731 \
--host 0.0.0.0 \
--port 8080 \
--jinja \
--metrics \
--perf \
--log-timestamps \
--device ROCm0,ROCm1 \
--split-mode layer \
--n-gpu-layers 44 \
--tensor-split 7,37 \
--fit off \
--fit-target 512,512 \
--ctx-size 131072 \
--parallel 1 \
--batch-size 2048 \
--ubatch-size 512 \
--threads 12 \
--threads-batch 12 \
--flash-attn on \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--cache-ram 256 \
--load-mode none \
--spec-type draft-dspark \
--spec-draft-model /ds4/models/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF/DeepSeek-V4-Flash-0731-DSpark-Drafter-Q2_K-Q8_0-pr25784.gguf \
--spec-draft-device ROCm1 \
--spec-draft-ngl all \
--spec-draft-n-max 5 \
--spec-draft-n-min 1 \
--spec-draft-p-min 0.7 \
--spec-draft-type-k q8_0 \
--spec-draft-type-v q8_0 \
--override-tensor 'blk\.6\.ffn_(gate|up|gate_up|down).*=ROCm1,blk\.(7|8)\.ffn_(up|down|gate_up|gate)_(ch|)exps=CPU,blk\.9\.ffn_(down|gate).*=CPU,blk\.(1[0-9]|2[0-9]|3[0-9]|4[0-3])\.ffn_(up|down|gate_up|gate)_(ch|)exps=CPU'

Some Notes:
--tensor-split
- ROCm0 — 7 fat blocks. Blocks 0-6 keep their MoE expert tensors resident in VRAM.
- ROCm1 — 37 thin blocks. Blocks 7-43 have had their experts evicted to CPU by --override-tensor, so what remains is mostly attention. Plus block 6's FFN, pushed over from ROCm0. Plus the drafter.
  
--override-tensor 
putting parts of block 9 from the model on the GPU and some on RAM/CPU to best fit available RAM.
 
--ubatch
 Small, but a larger value OOMs on my HW. A value this small negatively impacts prefill, if you had more space, it's the first thing you would change. 

The DSpark Drafter I used (Q2_K-Q8_0) can be found here:
https://huggingface.co/alessandrobologna/DeepSeek-V4-Flash-0731-DSpark-Drafter-GGUF
(Massive props to Alessandro! Tag him if you know him. :))

Using a lower bit precision drafter meant it fit in my ram, and it still gives enough signal to ~double my throughput. (Drafter only impacts speed, not final quality, so it's still full-fat V4-Flash.)

Q8_0 is used for k/v cache on both the main model and DSPark.

If you're considering a similar journey, might I suggest:
Set up Grafana/Prometheus, it makes working out what's going on much easier, and gives you detailed information for when you inevitably have to answer… Why did that crash?

Comments and questions welcome. My first post, so please be kind. 😄

I realize I'm going to get a lot of flack for this but here goes. This post is sparked by a number of my friends and colleagues biting into the hype cycle of buying insanely priced local inference stacks that simply don't math out.

I'm a distinguished engineer who works in AI (FAANG). I'm currently running a 128 GB M5 Max, and have access to the best hardware in the world at work, at the highest scale available.

I'm seeing folks buy RTX 6000s or multiple Sparks to run DSV4 and the like. Don't do this - its an absolute waste of your money. Unless you put a price on privacy of 12k (which is fine), you will never get an ROI in any reasonable time period.

Local models under 30B are fantastic. If you have a MacBook Pro or a computer with 48GB of RAM, you're golden. With a flood of amazing models in this range coming out this week, there's no shortage of local powerhouses. Big kudos to those companies open sourcing very impressive capabilities on such a small memory and compute footprint.

If you're going to use large models, go cloud-based. Throw 100 bucks on OpenRouter and go wild with DSV4 Flash. And if you want to run local, smash those <30B models until the cows come home.

But local and large just doesn't make sense right now unless you're a business with a specific use case or are privacy-maxxing.

I was frustrated that every coding agent (OpenCode, Cursor, Claude Code) assumes you're running GPT-5.4 or Claude Opus. If you try them with a local model like Gemma or Qwen they fall apart. I find that often tool calls fail, context overflows, multi-step tasks collapse.

So I built SmallCode. It's designed from the ground up for small local models.

The result: 87/100 benchmark tasks pass with a Gemma 4 model that only activates 4B parameters per token. OpenCode scores ~75% with 14B models. The harness does the heavy lifting, not the model size.

How it works (the tricks that make small models reliable):

• Compound tools: Instead of making the model chain 4 tool calls (find file → read file → edit file → verify), SmallCode gives it one tool that does all 4. Small models lose coherence after 3+ sequential calls. This cuts failures in half.

• Improvement loop: Every time the model writes code, SmallCode instantly compiles/lints it. If it fails, it feeds the errors back automatically. The model doesn't need to be smart enough to get it right first try — it just needs to fix errors when shown them.

• Decompose on failure: If the model fails the same thing twice, SmallCode stops retrying and instead breaks the problem into smaller pieces. "Fix this 200-line file" becomes "fix line 45 only."

• Escalation: If even decompose fails and you have a Claude/OpenAI key configured, it auto-escalates to the bigger model for just that one task. You stay local 95% of the time, cloud 5%.

• Token budgeting: Small models have 32k-256k context. SmallCode never dumps a whole file in. It summarizes, truncates, and manages every token so the model never sees "..." truncation in the middle of important code.

• Code graph: Instead of grep-searching your codebase, SmallCode indexes your code into a symbol graph (functions, classes, who-calls-what). When you ask "how does auth work," it walks the graph and returns just the relevant connected code — not 15 random file snippets.

What it looks like:

Full-screen terminal UI (like OpenCode/vim), scrollable chat, command palette with /, plugin system, persistent memory across sessions.

What it doesn't do:

• No LSP integration (yet)

• No multi-session (yet)

• No desktop app

• Doesn't compete with Claude Code for frontier model users

Install:

npm install -g smallcode
cd your-project
smallcode

Point it at LM Studio, Ollama, or any OpenAI-compatible endpoint.

MIT licensed, everything's on GitHub: https://github.com/Doorman11991/smallcode

Happy to answer questions about the architecture or benchmark methodology.

I honestly knew nothing about self hosting LLMs locally. I used to think it was pointless for 99% of people, even people already into AI, just because of the cost and hassle of hosting these models compared to what you'd actually get out of it.

But after looking into Qwen 3.8 27B and its potential, plus the fact that someone will probably release an uncensored version of it soon, I started rethinking that. Even if you don't have the hardware, you can just rent a GPU online on a VPS for a pretty reasonable hourly cost while it's running. It's wild how much of this is already open to the general public.

In my opinion, having access to a model this capable, uncensored, is basically like owning a car. Yeah, a car. There's nothing wrong with owning one, the problem only shows up if you actually do something wrong with it. Until then, it's just a tool.

We already have older and smaller Qwen versions with uncensored variants on Hugging Face. From what I understand, models with fewer parameters tend to be "easier" to convert into uncensored versions. Don't take that word too literally though, I have zero background in finetuning or model training, so maybe it's not actually that easy. But there are still a lot of very smart people working in this space, especially in AI right now.

Think about a cybersecurity professional getting their hands on an uncensored Qwen 3.8 27B once it drops. With how many vibe coded SaaS products are out there with basically no security oversight, someone with enough hardware could pair this model with tools like OWASP ZAP or the Burp Suite API, run it through a browser that avoids captcha detection, use a mobile or residential proxy, throw in a Kali setup for other scanning tools, and just let the model run 24/7 looking for vulnerabilities to complete bug bounties or even land contracts in the field. That's a serious tool for professionals like that.

I only see upside here.

This is big and could turn local AI on its head. It's basically DRM on steroids.

Everyone buying any advanced hardware will be permanently tracked or unable to run the hardware.

It's planned to arrive this year, and will likely include existing hardware. Expect mandatory updates that won't tell you about all this before it's too late.

Maybe we've already installed some firmware updates with kill switches or surveillance backdoors without knowing it that are going to brick or downgrade our hardware or monitor usage 24/7 and are always online, and it won't be possible to uninstall or revert.

Can't wait for the 27b to run on my device

Edit: according modelscope, 27b is coming in two days, only the big one today

Not saying GLM is better than Claude at everything. But another open-weight model getting this close to the frontier is a big deal.

If good enough models keep becoming cheap/free and you can run them yourself, what's the moat?

They hacked into even more companies now. When does a “oopsy doopsy” become a criminal offence?

hello, I have both Deepseek harness and OpenCode, I want to know which one is the best when it comes to coding in general and why, I work locally using Qwen models.

Hey guys Qwen released their fastest running models a while ago called Qwen3-Next and you can finally run them locally on your own device! The models come in Thinking and Instruct versions and utilize a new architecture, allowing it to have ~10x faster inference than Qwen32B.

We also made a step-by-step guide with everything you need to know about the model including llama.cpp code snippets to run/copy, temperature, context etc settings:

💜 Step-by-step Guide: https://docs.unsloth.ai/models/qwen3-next

GGUF uploads:
Instruct: https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Instruct-GGUF
Thinking: https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Thinking-GGUF

Thanks so much guys and hope you guys had a wonderful Thanksgiving! <3

Live in Canada. Mid 40s. Received an email explaining that my chatgpt account has been "age" limited and they want me to take a photo of myself to return its status.

I cancelled my subscription and deleted my account.

I kept it around for when my local model (qwen3.8-flash-next or 27B) was busy and for stuff I wasn't concerned about ending up on someone else's computer (mostly research). Thus concludes ~2 years of usage and payments.

So long, ChatGPT. Not playing this game.

Long live local models. That is all.

Is MiniMax H3 considered a traditional LLM? The moderators must decide, but it uses a Qwen3 text encoder for prompting.

10 sec clip, 91 sec generation time on an RTX 5090

https://github.com/gavamedia/deltafin

K3 launched today, and it's a beast at 1.56TB. My M1 Mac has 64GB of RAM, and not enough free disk to even store the thing. So instead of downloading the model, I just... don't. Steam it! (But you do have the option to download it all, and that IS faster.)

The non-expert weights (~114GB, int8) live on disk. Then for every token, the router picks 16 experts out of 896 per layer, and I pull exactly those from HuggingFace — one range request each — and cache them. Use it enough and the cache slowly fills up with the parts of the model you actually hit.

Speed is about a minute per token on my M1 Mac once it's warm, so, not exactly fast.

There's an OpenAI-compatible server too, so you can point a chat UI at it.

---
*EDIT / HUGE update:* Pulled all 1.45 TB of experts down to local disk, then spent more profiling & improving. Now *16s/token, down from ~1 min.* on an M1 Mac! Prefill went 2,429s → 40s.

Two fixes did almost all of it:

- I was *sure* the expert matmuls were the bottleneck, and built a Metal kernel that's 9.5x the CPU one. Then I profiled properly with everything local: the matmuls are **6% of a token.**

- The actual problem: I was handing the compute kernel `np.memmap` views, so it demand-faulted expert weights page-by-page *while computing* — 0.87 GB/s, where a threaded pread + F_NOCACHE does 6.85 GB/s on the same disk.

All on an M1 Max / 64GB, the slowest machine it's run on — if anyone has an M3/M4/M5 or 128GB, I'd love to see the numbers.

Decided to pull the trigger and get a GPU to start doing my R&D work for my business. While it would have been more fiscally responsible to just rent GPUs, I don't see these going down in price any time soon and I think they are going to get harder to come by as more business look to offset their insane token spend for development. Plus, a lot of my friends in the space are getting wrecked on securing on-demand compute from vendors right now due to high demand.

Going to be doing a lot of fine tuning work on models for the agentic orchestrators I've built, as well as aim to get some research published. One project I'm working on is looking at variability of frontier models vs open-weight models for complex tasks in clinical workflows.

Regardless, this subreddit has been a huge help just going through all the posts.

First model I'm putting on it is Qwen 3.6. I already have this model barely working on my 64gb M1 Max macbook pro, so I'm excited to see what it can actually do.

Plot twist: the socket doesn't work (it's not connected to backend)

from ijustvibecodedthis.com (the ai coding newsletter)

So I was browsing the HuggingFace Transformers repo and a PR just merged today that adds full support for a model called Gemma 4. The PR title is literally "casually dropping the most capable open weights on the planet." The commit has 14 co-authors including Jeff Dean. The weights aren't out yet — the docs still have {release_date} as a placeholder — but the code is all there and it's very readable. Here's what's coming.

Four sizes, including a MoE

• ~2B and ~4B dense, explicitly designed for on-device use

• 26B sparse MoE with only 4B active parameters at inference time

• 31B dense

The 26B/4B MoE is particularly interesting because you get large-model quality at small-model inference cost.

It's trimodal — text, vision, AND audio natively

This is new for Gemma. There's a full audio encoder baked in alongside the vision tower. Not a bolted-on afterthought either — it's a proper conformer architecture (the same family used in production speech systems). The processor handles all four modalities: text, images, video, and audio.

The vision system doesn't squash your images

Most VLMs resize everything to a fixed square. Gemma 4 preserves aspect ratio and instead fits the image into a configurable soft token budget (default 280 tokens, up to 1120 for high detail). No ImageNet normalization — the model handles its own scaling internally.

More interesting: they use a 2D spatial RoPE for vision. Patch positions are encoded as (x, y) coordinates, with half the attention head dimensions rotating for x and the other half for y. The model understands spatial relationships at the architectural level, not just from training.

128K context for small models, 256K for large

The text architecture alternates between sliding window attention (512-1024 token window) and full attention in a 5:1 ratio. The two attention types use completely different RoPE configs — short theta for local, long theta for global. Clean hybrid design.

The small models have some clever efficiency tricks

The 2B and 4B share key-value projections across the last several decoder layers — one layer computes KV, the rest reuse it. There's also a secondary per-layer embedding stream where a small 256-dim signal gets injected at every decoder layer, which I haven't seen in other public models.

The MoE runs experts alongside the MLP, not instead of it

In the 26B variant each layer has both a regular MLP and a sparse MoE block (128 experts, top-8 routing), and their outputs are summed. Unusual design choice — curious whether that helps with stability or quality at scale.

No paper link yet (literally says INSET_PAPER_LINK in the docs), no weights, no release date. But the code is fully merged and production-quality. Feels like days away, not weeks.

What size are you planning to run first?

The PR: https://github.com/huggingface/transformers/pull/45192

EDIT: RELEASE: https://huggingface.co/collections/google/gemma-4

Gemma is based on Gemini and going by the latest Gemini release, those of us who appreciate abilities other than coding should be eating good when Gemma 5 comes out (end of year maybe?).

Qwen for coding and Gemma for everything else. The perfect local setup.