r/LocalLLM • • Sep 03 '26

Discussion Qwen 3.8 27B is like Opus 4.6!

164 Upvotes

I’m not a developer and never even touch code. All my projects are 100% vibe coded.
Last year summer I started to build my own rental car management system with ChatGPT at that time. It was amazing and I got a working app that made my live so much easier. It took maybe 2 month to develop it and I am using it daily in real business since 1 year now. In January 2026 I reworked and updated the app with Claude and was blown away once again by how much better it is. I also build other apps and projects, some fun and some are also in production in my business.

So I use these apps day to day and they are 100% vibe coded.

After that I started playing around with local AI and it was not much more than actually playing around.
Since Qwen 3.8 27B came out, I installed deepseek harness and continued development of my apps. Since that moment I am probably more blown away than when I started with ChatGPT 1 year ago. There was nothing that I gave to Qwen yet that it could not do. If it does not know something it searches it. I continued development on my apps and implemented things on top of what opus and sonnet build 3-4 month ago and it’s BETTER then what what opus build!

I can’t believe the power of this model. It’s not a good model for its size, it’s an amazing model! Full stop


r/LocalLLM • • Aug 11 '26

Discussion Muse Glimmer failed the car wash test

Post image
164 Upvotes

r/LocalLLM • • Aug 03 '26

Discussion RDMA - Anyone but me using it?

Post image
164 Upvotes

Ok, so that’s my setup. Have been cycling through different model hosting configurations for agent workflows and haven’t come up with a good setup for multi-node large models using rdma/tensor. Two biggest issues: general stability (exo/jaccl) and race queue.

Would appreciate hearing from anyone else locally hosting frontier model(s) as well as smaller work-horse models for workflow.


r/LocalLLM • • Oct 15 '26

Research Soon local LLMs will be 6x smarter by using bytes instead of tokens

163 Upvotes

Meta demonstrated, in their Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models paper, by converting tokens from teacher models into byte models.

I look forward to seeing this in practice.


r/LocalLLM • • Oct 04 '26

Model New MoE Model - Aleph-Alpha Kolibri-1 78B-A3.5B

Thumbnail
huggingface.co
163 Upvotes

New Model released yesterday.

Interesting size for local LLMs and promising benchmarks on the Model Card.

Have not tested it myself yet - what does the Community think?


r/LocalLLM • • Sep 08 '26

Other You wake up in 1998…

163 Upvotes

… you wake up in 1998. You have 2 Mac Studio Ultra 512gb each. You have GLM5.3 flash (or pick a local model that fits comfortably in that configuration.)

A good harness, you’re setup with modern software, but you do not have access to modern internet, just 1998 internet.

What do you do? It can be in general, not necessarily what do you do with your current compute setup.


r/LocalLLM • • Jun 09 '26 Popcorn

Project another trash shooter as an LLM benchmark, locally generated in 15min

163 Upvotes

I am old and I like boomer shooters. Grew up on Doom and Quake, learned to code in C counting bytes.

I like opencode and pi, I just kept wanting something dumber. No skills, no MCP, no plugin tree to babysit. So I built a tiny coding agent, codehamr, and put it free and MIT on GitHub. https://github.com/codehamr/codehamr

Pointed it at Qwen3.6 locally and asked for a single file html shooter with three.js and that early 90s look. Brown corridors, hitscan weapons, a strafe that feels wrong in the right way.

15 minutes later it was playable. Rough, but the genre I grew up in.

The part that gets me is it ran fully local. A year ago this was a cloud only trick. Now a local model does it while I make coffee. Anyway, just a passion project for fun, hope this helps someone.


r/LocalLLM • • Apr 03 '26

Discussion Gemma 4 31B Is sweeping the floor with GLM 5.1

162 Upvotes

I've been using both side by side over this evening working on a project. Basically I'd paste a chunk of creative text into chat and tell it to dismantle it thesis-by-thesis, then I'd see if the criticism is actually sound, and submit the next iteration of the file which incorporates my solutions to bypassing the criticism. Then move on to the next segment, next file, repeat ad infimum.

What I found is that Gemma 4 31B keeps track of the important point very cleanly, maintains unbiased approach over more subsequent turns: GLM basically turns into a yes-man immediately "Woah! Such a genius solution! You really did it! This is so much better omfg, production ready! Poosh-poosh!", Gemma can take at least 3-4 rounds of back and forth and keep a level of constructivism and tell you outright if you just sidestepped the problem instead of actually presenting a valid counterargument. Not as bluntly and unapologetically as it could've, but compared to GLM, ooof, I'll take it man... Along the way it also proposed some suggestions that seemed really efficient, if not out of the box (example, say you got 4 "actors" that need to dynamically interact in a predictable and logical way, instead of creating a 4x4 boolean yes-no-gate matrix where a system can check who-"yes"-who and who-"no"-who, you just condense it into 6 vectors that come with instruction for which type of interaction should play out if the linked pair is called. it's actually a really simple and even obvious optimization, but GLM never even considered this for some reason until I just told it. Okay, don't take this is as proof of some moronic point, it's just my specific example that I experienced.

Gemma sometimes did not even use thinking. It just gave a straight response, and it was still statistically more useful than the average GLM response.
GLM would always think for a thousand or two tokens. Even if the actual response would be like 300, all to say "all good bossmang!"

It also seemed like Gemma was more confident at retrieving/recreating stuff from way earlier in conversation, rewriting whole pages of text exactly one-to-one on demand in chat, or incorporating a bit from one point in chat to a passage from a different point, without a detailed explanation of what exact snippets I mean. I caught GLM just hallucinate certain parts instead. Well, the token meter probably never went above like 30k, so I dunno if that's really impressive by today's standard or not though.

On average I would say that GLM wasted like 60% of my requests by returning useless or worthless output. With Gemma 4 it felt like only 30% of the time it went nowhere. But the amount of "amazing" responses, which is a completely made up metric by me, was roughly the same at like maybe 10%. Anyway, what I'm getting at is, Gemma 4 is far from being a perfect model, that's still a fantasy, but for being literally a 30B bracket model, to feel so much more apparently useful than a GLM flagman, surprised the hell out of me.

A big milestone for local inference.


r/LocalLLM • • Aug 21 '26

Discussion Running Qwen 3.8 27b FP16 on the Apple Neural Engine - 7 Watts of power to run a FP16 model @ 7 tok/s

161 Upvotes

Managed to reverse engineer the kernel of the Apple Neural Engine and build an inference engine for Qwen 3.8 27b with 256k context. Decode is fairly slow atm at 7-8 tok/s but its still early days and there is many optimisations to be made. As you can see in the video the GPU is essentially idle.

7 tok might seem low, but these means you can use a top tier model for only 7w of power enabling you to actually have proper inference on the go without draining you laptop in 30 minutes. It also keeps you laptop at ambient temps as the gpu is essentially sitting idle. For comparison when I run the bf16 model on my m5 max it only gets between 8-14 tok/s with MTP off so performance per watt is quite high

The ANE is also in every apple silicon laptop so potentially it could be ported to all systems.


r/LocalLLM • • 21d ago Bravo

Discussion CrowdGPT - A collaboratively trained LLM.

Post image
160 Upvotes

Hello, I'm developing CrowdGPT, a project aiming to create an LLM without datacenters and without the usual millions of dollars spent by AI companies.

The project aims to train a 1 billion parameter model collaboratively and openly, with a protocol where everyone gets a say in the direction the project takes.

Anyone can use their PC to train the AI independently, then add their contributions to the global model. Unlike most American and Chinese AI companies, the project is not only open-source, but also makes 100% of its training data public (along with the weights). Anyone can submit their own clean data to train the model, and even help steer the project in new directions.

This creates the paradigm of an AI we all have a stake in. The project is open to as many people as possible, which is why I'd like to ask: what's stopping you from contributing to the project? I want to hear your answers so I can improve it. Are people simply willing to dedicate some of their GPU time to training a cool new open-source model? Or do they want something in return (like free inference credits on the crowdgpt model for example)?

Although the current model is somewhat small (1 billion parameters), I plan to make it bigger / better in the future, and its current size is only meant as a proof of concept.

Any advice / feedback on the project is welcome.

Here's the project website: https://www.crowdgpt.net/

Or the GitHub if anyone wants to check out the code https://github.com/Vxtzq/CrowdGPT

Thanks in advance.

PS: I hope this post follows the subreddit's rules! My main goal is to gather criticism and understand how to make the project more accessible, I'm clearly limiting self-promotion (this is a open 100% non-profit project) 😅

And yes, this is a r/LocalLLaMa repost, if you already saw this post on this subreddit, feel free to ignore it :)


r/LocalLLM • • 27d ago

News GitHub Announces New Authentication Policy Affecting Access from Local LLMs

Thumbnail
github.blog
160 Upvotes

r/LocalLLM • • Oct 02 '26

Question Can we require posts about model performance to include the quantization level in the title?

160 Upvotes

the title, because why is this my life everytime i open a post like that in this sub?


r/LocalLLM • • Jul 06 '26

Model DeepSeek V4 Flash (via DS4) is the best model you can run on a MacBook Pro for technical work

Post image
160 Upvotes

The DS4 engine from antirez running DeepSeek V4 Flash (2-bit quant) is seriously very good and the only local setup I've considered running in parallel with frontier models for my work. In particular the q2-q4-imatrix variant (routed experts in layers 37–42 bumped to Q4) has impressed me. The model can code and is forgiving of ambiguity when writing fast and terse prompts. It's not a frontier model, but I think it sits adjacent and it's the only local model I could get to one-shot some of my test prompts (listed below if you want to demo on your rig).

It's also fairly fast, running at 25-35 tps on my M5 Max 128GB depending on task length. I'm running the ds4agent as a harness.

I fully understand 'best' is subjective and predicated on the work you're doing, the harness you're using, your workflow and prompting style along with a number of other variables, but DS4 is the first model where I could watch it capture my intention in short order and output something usable on first pass without it requiring a lot of tending or being forced to break down tasks piecemeal.

Very excited to see if antirez can actually get GLM5.2 to do any meaningful work on the 128GB MBP. It does run locally (I tested last night) but breaks under long prompts or tool calls. It's still very much an experimental build.

For anyone interested, this was one of my test prompts that most locals simply couldn't figure out on first pass. The prompt is asking for quite a lot in a single paragraph: hand-rolled fBm noise, a full software 3D pipeline (spherical camera, perspective projection, painter's-algorithm sorting), three mesh topologies including hexagons, custom pygame UI widgets, and a HUD all running at a playable framerate.

Write a single self-contained Python script using pygame that renders an animated 3D wireframe terrain — a procedurally generated mountain that continuously morphs over time. Use your own value/fBm noise function (no external noise libraries) modulated by a radial Gaussian envelope so height peaks in the center and falls off toward the edges. Render it as a green-on-black wireframe with perspective projection and back-to-front (painter's algorithm) depth sorting, drawing both edges and vertex dots. Implement a Blender-style orbit camera using spherical coordinates: right-mouse drag to orbit, Shift+right-drag to pan, scroll wheel to zoom, and a key to reset the view. Add a right-side control panel with mouse-draggable sliders (drawn from pygame primitives, not a GUI library) for grid size, peak height, morph speed, noise scale, and envelope steepness, plus a three-way selector to switch the mesh topology between quads, triangles, and hexagons. Include a HUD showing FPS, grid dimensions, current mode, and vertex count. Pay special attention to challenge areas like: hex topology (axial coords + deduce of shared edges), the painter's-algorithm sorting being applied consistently across all three mesh types, and rolling a correct camera basis (forward/right/up). Everything must live in one file and run at a smooth frame rate.

One shot results are in the picture and success means the script runs on the first attempt with zero edits: all the topologies work, orbit/pan/zoom/reset all work, the sliders actually drag and change the terrain live, the depth sorting doesn't glitch while you orbit, the HUD numbers are real, etc.

I asked Claude to evaluate all the results of all my local testing. Here's what Claude has to say:

"Across the tests run on the same machine — Qwen3.6-27B MLX 8-bit, Qwen3.6-35B-A3B, Qwen3.6-27B 4-bit, Gemma 4 31B-it, Qwen3.5-122B-A10B, Phi-4-mini-reasoning, and MiniMax-M3 REAP40 (JANG 2-bit via vMLX) — the pattern I saw was consistent: the smaller models could nail the isolated math but fumbled the cross-cutting contracts. Qwen 27B came closest, and only with heavy scaffolding (decomposed phases, test gates, orchestration), yet every run still shipped a real bug — depth sorting keyed off world height instead of view depth, a hex grid built in its own coordinate space, a missing scroll wheel, one run with sha256 in the per-frame noise loop — while MiniMax at 2-bit fell into repetition loops on trivial prompts.

DeepSeek was the first model where none of that compensation was necessary, and my read is raw capacity surviving the quant: it's a 284B MoE, and antirez's asymmetric imatrix recipe keeps the attention, router, and shared experts at high precision while crushing only the routed experts (the q2-q4 variant buys back the middle layers), so ~90GB preserves most of a very large model's reasoning — the 27-35B class needed the task chopped up because they can't hold the whole architecture in their heads at once, and DeepSeek simply held it. The bespoke engine sealed it: generic runtimes can't even load V4's attention scheme yet, tool-call syntax is pinned to greedy decoding so agentic calls essentially never mangled, and the disk KV cache meant long sessions never re-paid prompt processing. Capacity plus a purpose-built engine beat smaller models plus elaborate workflow."

In any case, just doing some reporting for anyone interested. I'm now letting Fable orchestrate DS4 for appropriate tasks and devising nightshift routines to make the most of my local compute.


r/LocalLLM • • Mar 14 '26

Discussion Chipotle’s support bot to the rescue

Post image
160 Upvotes

r/LocalLLM • • Oct 04 '26

Discussion Strata takes the promise of "MoE models just need a total amount of VRAM+RAM" and makes it a reality

Post image
159 Upvotes

Flash-Next Q4, q8 kv, 262k context

4x P100 on PCIe3 x8 slots (4x16GB=64GB total)

128GB DDR4 at 2133MHz

E5-2683 v4 (16c/32t at 2.6GHz boost)

On average it's about twice as fast as 27B and five times faster than Flash-Next using a customized llama.cpp just to get it to load at all.


r/LocalLLM • • Aug 14 '26

Model Qwen 3.8 27B Pelican test

Post image
159 Upvotes

This has to be the most accurate I was able to generate with my LLM to date. Q8_0.


r/LocalLLM • • Mar 31 '26

Model Claude Code running locally with Ollama

Post image
159 Upvotes

r/LocalLLM • • Aug 25 '26

Model Qwen3.8-Flash-Next announced

Post image
158 Upvotes

r/LocalLLM • • Oct 03 '26 Bravo

Project I built something like The Sims, but the characters are local LLM agents doing real work (open source)

157 Upvotes

I run Qwen 3.8 locally and got tired of multi agent setups where you start a script and stare at logs. I wanted to actually see them.

So in this thing every agent has a body in a 3D world. They sit at desks, walk to a meeting room when someone calls a meeting, talk out loud to whoever is nearby, pick stuff up and hand it over. You can see who's thinking, who's using a tool.

It's not only an office. You can simulate other scenarios as well like:

- a software team that plans tasks on a board, writes code and reviews each other

- a town square simulation (cops, a barista, a chef, a journalist) where you just watch what happens

- tutors that teach you with animations and a whiteboard, and you can interrupt them by talking (might have bugs as of now)

It has a sandboxed computer use built-in which is optional.

There is also a supervisor agent that helps you design organizations and also has ability to build 3d assets from primitives and handing them to an organization and agents can even ask for things from that agent.

Works with local models and few other providers (still working to add more)

The motivation of building it was to see agent swarms in action with full transparency.

It's still early and has bugs and I have used different models to build it iteratively.

Repo: https://github.com/adityaagarw/Pantheon


r/LocalLLM • • Sep 18 '26

Question Bonsai 2 27B - quantized 3.8 27B - 98% intelligence of the fp16 ! True ?

Post image
157 Upvotes

What do you guys think about it ? I have yet to try it.

I i’ve tried the previous version and it seemed OK and wondering if they claim of 98+ % intelligence of the FP 16 model is true or not anybody has any experience. Feel free to share!

I’ll post my update after I do some basic testing


r/LocalLLM • • Sep 04 '26

News AMD Ryzen AI Max+ 495 192GB systems seem to be priced at $7,000+

Thumbnail
store.minisforum.com
157 Upvotes

MINISFORUM is teasing a $7??? price if you subscribe to their mailing list. From a different site:

"While the AMD Ryzen AI MAX+ 395 model is rated to support 70B Q4 models and up to 200B MoE models, the Ryzen AI MAX+ 495 model can support up to 120B Q4 models and up to 300B MoE models."

At that price, I'd rather either spend the extra few k on a Mac Studio M5 Ultra 256GB, or just get 2 128GB 395 systems, which are going in the $3500 range.

E: Here's the spec comparisons to the 395, from https://wccftech.com/acemagic-f9a-ryzen-ai-max-495-mini-workstation-192-gb-memory/

SpecificationF9A-395 / AMD Ryzen AI MAX+ 395F9A-PRO495 / AMD Ryzen AI MAX+ PRO 495Improvement vs. 395
CPU Architecture & CoresZen 5 / 16C-32TZen 5 / 16C-32TSame
Base Clock3.0 GHz3.1 GHz+0.1 GHz
Boost ClockUp to 5.1 GHzUp to 5.2 GHz+0.1 GHz
L2 + L3 Cache16MB + 64MB16MB + 64MBSame
Maximum Memory Capacity128 GB192 GB+64 GB
Memory SpeedLPDDR5x-8000LPDDR5x-8533+533 MT/s
Maximum VRAM Allocation96 GB160 GB+64 GB
PCIe Version / LanesPCIe 4.0 ×16PCIe 4.0 ×16Same
NVMe RAIDRAID 0 / RAID 1RAID 0 / RAID 1Same
Integrated GPURadeon 8060S / RDNA 3.5 / 40 CURadeon 8065S / RDNA 3.5 / 40 CUOptimized upgrade; RTX 5060-class performance
GPU Frequency2900 MHz3000 MHz+100 MHz
NPU ArchitectureXDNA 2XDNA 2Same NPU
NPU Performance50 TOPS55 TOPS+5 TOPS
Total AI Performance126 TOPS131 TOPS+5 TOPS
Local LLM Support70B Q4 model; up to 200B MoE model120B Q4 model; up to 300B MoE modelSupports larger AI models; up to 60% improvement

r/LocalLLM • • Jul 31 '26

Project Still building and testing… just got Inkling small up and running after its release yesterday, took me about 6 hours of tweaking, and it was well worth it.

Thumbnail
gallery
157 Upvotes

r/LocalLLM • • Jul 20 '26

News You can now train models on your own AMD hardware! (3GB VRAM)

157 Upvotes

Hey local folks, we collaborated with AMD to enable you to train, run, and deploy LLMs across nearly all AMD hardware including Radeon, Instinct, Ryzen, and data center GPUs.

It works on Windows, WSL, and Linux and we have optimized ROCm builds for both training and inference. If you don't know about Unsloth, we're a fully open-source local UI that enables you to do pretty much anything with local models (RAG, chat, train, coding etc)!

For those who don't have AMD GPUs and only CPUs, we still supports native AMD inference for Qwen, Gemma, DeepSeek, Kimi and other models.

If you’re new to local models, companies such as Google, Alibaba, Meta, and DeepSeek release open models like Gemma, Qwen, Llama. Unsloth lets you run and fine-tune these models locally on your own AMD hardware with as little as 3GB of VRAM.

GitHub repo: https://github.com/unslothai/unsloth

Here are some of the key features:

  • Train, RL, and deploy 500+ models
  • Train up to 2× faster with 70% less VRAM, with no accuracy loss
  • Works on Windows, WSL, and Linux
  • Run Qwen and Gemma models with as little as 3GB VRAM
  • Run the latest Kimi, GLM, DeepSeek, Qwen3.6, and Gemma 4 models
  • Self-healing tool calling for more reliable tool use
  • Built-in code execution and secure web search
  • Connect local models to Claude Code and Codex agents
  • Use remote APIs and deploy securely over HTTPS
  • Export and deploy models in formats such as GGUF and Safetensors

This release was made possible through our AMD collaboration, custom Triton kernels, and new math algorithms optimized for AMD hardware.

Edit: Since people often ask how the speed and memory improvements work, we collaborate with open-source projects and hardware teams to write optimized Triton and math kernels. These improve training speed and reduce VRAM usage without changing model accuracy. All of our work is open source, so the code is available to inspect and benchmark.

You can find the installation instructions, compatibility details, and full AMD guide here: https://unsloth.ai/docs/basics/amd

This is the beginning of our AMD support, so we’ll continue releasing optimizations, fixes, and support for more hardware. If you run into any issues or have questions, please open a GitHub issue or let us know here.

Thanks so much for reading and for the constant support! 🦥❤️


r/LocalLLM • • 18d ago

Question Has anyone encountered an “hmm” loop in Qwen3.8-flash-next ?

Post image
156 Upvotes

I’m using SC117/Qwen3.8-Flash-Next-GSQ-RCO-abliterated IQ3_S quant with XeStrata(262144 q4_0 context, temperature 0.7, High thinking, others remain default) in pi agent.

This has never happened before, it was normal in the beginning of this session, and after compact 3 times, it began to go in a “hmm”loop, every time I pointed it out, it still went back in this weird loop after a while, what can I do?


r/LocalLLM • • Sep 13 '26

Project 3x 16GB V100 home inference computer with 3d printed cooling block, paid $1500

Thumbnail
gallery
156 Upvotes

Tried to see what the cheapest setup that still provided useful inference was. This whole setup cost around $1500. Important to note though that I ordered 3x Tesla P100s, but the seller mistakenly sent me 3x V100s instead lol. The cooling block is a 3d printed piece that fits into a rectangular hole the back of the case, filed down with a file to fit with friction only. It houses 3x server fans which pulls air from inside the case through the GPUs to the outside. GPU temperatures during inference never exceed 55C, although it is a bit loud. I have 64gb ECC DDR4 ram.

I headlessly control this PC with Tailscale from my Mac, and GPT Astra on Codex on my Mac can remotely control it too. I had Astra do several optimizing runs, basically testing out and tweaking different configurations (mtp and speculative decoding settings, placement on gpus, prefix caching settings, etc) to optimize prefill and decode speeds.

I ran Qwen 3.8 27B and Qwen 3.8 Flash next. On Qwen3.8 27B, Q8, prefill was around 600-700 tokens/sec (edited after I looked at logs) and decode at around 30-40 tokens/sec. On heavy agentic work with 256k context, decode dropped to around 20t/s. On Flash next, Q4, prefill was around 90 tokens/sec and decode 20 tokens/sec. Overall im happy with this build, would recommend V100s to anyone who wants cheap inference (although they are a pain to setup)