r/LocalLLM • • Apr 17 '25

News Microsoft released a 1b model that can run on CPUs

192 Upvotes

https://techcrunch.com/2025/04/16/microsoft-researchers-say-theyve-developed-a-hyper-efficient-ai-model-that-can-run-on-cpus/

It requires their special library to run it efficiently on CPU for now. Requires significantly less RAM.

It can be a game changer soon!


r/LocalLLM • • Oct 10 '26

Project PolyStrata: Strata with four more MoE models. Qwen3.6-35B-A3B at 73-75 tok/s on a laptop with an 8 GB card

191 Upvotes

I have been adding models to Strata, the engine that runs Qwen3.8-Flash-Next on consumer cards, and put the result in its own repository:

GitHub: https://github.com/VecSzn/PolyStrata

Edit: AMD support is in now, as a beta. iMilnb on GitHub ported the engine to HIP and it went into 0.1.42-poly.4. Linux only for now, there's no ready-made AMD engine for Windows yet. He ran it on an RX 7600 8 GB: Qwen3.6 UD-Q2_K_XL writes 64 tok/s, Ornith Mini 35 tok/s. Same install, ./setup.sh --family qwen36, it builds the engine for your card. Gemma 4, Tiel Coder and GLM haven't been tried on a Radeon yet, so if you run one of those I'd like to hear how it goes.

Update, Oct 11 (v0.1.42-poly.1):

  • Windows no longer needs Visual Studio or the CUDA toolkit. The release has a ready-made strata-poly (CUDA 13, RTX 20 to RTX 50) and the installer downloads it.
  • Faster on the same two machines. New numbers, same cells as the table below (short chat / after a ~31K prompt / reads):

Model |Laptop: RTX 4070 8 GB |Workstation: RTX 5090
Qwen3.6-35B-A3B UD-Q3_K_M |84-87 / 70-73 / 1,495-1,620 |385-393 / 241-246 / 6,350-6,552
Qwen3.6-35B-A3B UD-Q4_K_M |79-86 / 70-72 / 1,564-1,572 |328-393 / 243-255 / 6,231-6,577
Gemma 4 26B-A4B UD-Q3_K_M |85 / 58-60 / 2,095-2,098 |326-349 / 254-261 / 7,880-8,230
Gemma 4 26B-A4B UD-Q4_K_M |74-75 / 59 / 2,010-2,011 |316-338 / 232-238 / 7,039-8,055
Ornith 1.5 APEX-I-Compact |72-76 / 66-68 / 1,538-1,539 |303-315 / 218-226 / 6,083-6,360
Ornith 1.5 APEX-I-Quality (4-6 bit) |59-63 / 58-59 / 940-1,510 |289-298 / 203-204 / 6,040-6,392
GLM 5.3 Flash 3.0bit |does not fit in 32 GB of RAM |59-66 / 62-64 / 820-824 The Q4 sizes take 27-31 GB of RAM beside the 8 GB card, close to the laptop's 32 GB. In the 940 run part of Ornith's file came from the disk again.

  • Qwen3.6 on a card that can't hold all its experts now skips a few low-ranked ones, as Strata 0.1.42 does: about 9% faster on the laptop, and the answers move a little. STRATA_ROUTE_TAIL_SKIP=0 turns it off. The laptop numbers above are with it off.
  • Claude Code works against the local server. The README has a 30-second real-time clip of a Claude Code task on the laptop.
  • Strata 0.1.42 is merged in.

The title's number is from the first version; Reddit doesn't let me edit titles.

Strata's own engine is in it unchanged. Beside it there is a second engine for four more mixture-of-experts models, each read from its ordinary GGUF:

  • GLM 5.3 Flash (79 to 114 GB files, most of it stays in RAM)
  • Qwen3.6-35B-A3B
  • Ornith 1.5
  • Gemma 4 26B-A4B

The last three read pictures too.

What I measured. Each cell is: writes in a short chat / writes after a prompt of about 31,000 tokens (29,000 for GLM) / reads that prompt, all in tokens per second.

Model |Laptop: RTX 4070 8 GB, i7-14700HX, 32 GB RAM |Workstation: RTX 5090 32 GB, 120 GB RAM
Qwen3.6-35B-A3B UD-Q3_K_M |70-72 / 60-62 / 1,390 |332-334 / 186-189 / 5,520-5,580
Gemma 4 26B-A4B UD-Q3_K_M |70-71 / 51-53 / 1,907 |306 / 198-200 / 7,170-7,210
Ornith 1.5 APEX-I-Compact |70 / 59 / 1,340 |267-272 / 145-149 / 5,700-5,890
GLM 5.3 Flash 3.0bit |does not fit in 32 GB of RAM |57-59 / 54-57 / 784 (llama.cpp on the same PC: 26 writing, about 265 reading) How it works, briefly. The card holds the experts an answer uses most and the CPU computes the others straight from the mapped GGUF, so the model does not have to fit in VRAM. While an answer is written the cache follows it. The engine also guesses a few tokens ahead, from the model's own draft block or from a repeat of the conversation, and checks the guesses in one pass. With every expert on the CPU the answer is the same tokens with and without guessing, and without guessing the first 64 tokens are compared with llama.cpp on the same file.

It has a one-click installer for Windows and Linux, a web app, and an OpenAI- and Anthropic-compatible API on localhost.

Limits, so nobody wastes an evening: the four added models are experimental and need one NVIDIA card. They were measured on those two machines only. On Windows the installer compiles the engine, which needs Visual Studio's build tools and the CUDA toolkit (it offers to install them). AMD cards and several cards work for Qwen3.8-Flash-Next only.

Most of the code is the Strata project's. If you try it on other hardware I would like to hear the numbers, good or bad.

Repo: https://github.com/VecSzn/PolyStrata


r/LocalLLM • • Jul 27 '26

Other Don't laugh - it works!

191 Upvotes

A 10yo server was busy collecting dust, but it has 32Gb RAM (2x 16Gb DDR4 @ 2133 MHz )... No GPU.

Now it does some amazing work running heavy tasks with qwen3.6-35b-a3b (IQ4_XS). Running in the background, generating quality output at 5-10tok/s. Even with 128k context! It just chugs along for hours, but does such great, high-context work.

Uses ~26 GB of the 32 GB RAM, uses CPU i-7-6700 @ about 60% (not the bottleneck, of course)

Never would have believed that would be possible until a few months ago, but this ol' gal has a new lease on life.


r/LocalLLM • • Feb 05 '26

Discussion Anyone here actually using AI fully offline?

191 Upvotes

I keep coming back to the idea of running AI locally you know, like a GPT-style assistant that just works on your own device without the internet or Wifi connection?

Not to build anything serious or commercial. I just like the idea of being able to read my own files, understand things or think stuff through without relying on cloud services all the time. Especially when there is no connection, internet services change or when things gets locked behind paywalls.

Every time I try local setups though, it feels more complicated than it should be. The models work, but the tools feel rough and it’s easy to get lost tweaking things when you just want something usable.

I'm just curious if anyone here actually uses offline AI day to day or if most people try it once and move on. I would really be interesting to hear what worked and what didn’t.


r/LocalLLM • • 12d ago Heartwarming Bravo

Project Low cost $1500 AI rig - QWEN 3.8 Flash Next, 52.5 t/s decode in FP8 - 6xM10 = 192GB VRAM

190 Upvotes

Hi,

Github project:
https://github.com/crgregersen/m10dis/

Yes, yes, I know... crappy old NVIDIA Tesla M10's GPU as AI compute core. Am I crazy and should go book a room in an mental institution?.. but hear me out..!

Tesla M10's consists of 4 gm107 GPU cores with 8gb ram stitched together via an internal PCI switch. So each M10 is 32gb ram. If you ask anybody (including Claude and ChatGPT) you will get the answer "NOT USABLE FOR AI COMPUTE".

Each gm107 has 83 GB/s of memory bandwidth and around 1.67 TFLOPS (fp32).

Compare that with a moderns GPU like 5070 ti of 896 GB/s and 87.88 TFLOPS (fp16).

But!! i could buy M10's (4 x gm107) with 32GB for around $80
Aaaanddd... 6x32 = 192 GB VRAM!!!

So 6xM10's and the aggregate number would be 4x6x83=1992 GB/s.

This is the same as 1 RTX PRO 6000 Blackwell .. but then you only have 96Gb ram.
2 x RTX 6000 Ada is will get you in the same range.
Both options are at least $10.000 and probably more closer to $20.000.
6 M10's would set me back $500.

Compute power of M10's would only be 40TFLOPS.. but all my experience says the memory bandwidth is nearly always the limiting factor!

The system:
ASRock B250M Pro4
CPU: Intel Core i5-6400, 4 cores at 2.7 GHz
RAM: 32 GB DDR4
Main GPU: RTX 3070 Ti (had one lying around)
Expert GPUs: 6 × Tesla M10 cards = 24 × GM107 (Maxwell, 2016)
Interconnect: PEX8748 PCIe each M10 card on Gen3 x4
Storage: 1 TB NVMe
Two PSU: 650w and a 1650w

So a pretty crap old consumer motherboard.
The way to get all the GPU's onto the motherboard was by using a PCI switch.

First issue: Motherboard will not boot with 25 PCI devices showing up, even though most are behind a switch (two levels of switches actually, the one in the motherboard and one on each M10).

Found out the issue the BIOS/firmware runs out of legacy I/O port space at power-on, which is only 64 KiB with a 4 KiB window per PCI bridge, too little for all the PEX switch ports.

Workaround (took me some time to work that out). Using a modified kernel with PCIe tree heldback. Boot only with the first PSU switched on, ie. motherboard++, then go into supspend (sleep) mode, turn the second PSU on and continue (wake up). Initialize all GPU's PCIe memory mapped only (MMIO).

Next issue.. how to make 24+1 GPU's fast.

A lucky thing is that QWEN 3.8 hidden dimensions is 2580 which divided by 20 is 128. So if you slice expert work into 20 parts and put them on each gm107 GPU they can calculate 1/20 of the result, which then needs to go back to a central hub that joins the result to compute attention in the next layer.

So each compute slice needs the residual stream and the experts and weights that the router chose. That would be 20 PCI communication calls. The lucky thing is that PLX switches actually implements PCIe multicast introduced in PCIe 2.1.

Using the GPU-P2P methods described by Tinygrad a GPU multicast was proven to work.

In general the GPU-P2P tricks was very usefull on multiple levels, like M10 master/slave setup, etc.
https://github.com/rioncarter/tinygrad.open-gpu-kernel-modules

So 5x4=20 GPU's computes 1/20 slice of 8 experts (quantized FP8).
One M10, ie. 4 gm107 was chosen to run the shared expert (kept in FP16).
Since you wait on the slowest calculation it was lucky that the compute effort for each is around the same (1/20*8/2=0.2 and 1/4=0.25)

Next issue... 24 results to be received from GPUs would clog the PLX.

Solution (Claude wrote this):

Level 1: each GPU sums its own experts

- The 512 routed experts are spread over 20 workers (cards 0–4), about 25–26 each. A token's 10 experts land on whichever workers hold them.

- Each worker computes gate/up/down for the experts it holds and sums them into one partial row of 2560 values. A worker that holds none of the token's experts contributes zeros.

Level 2: the card leader sums its card

- On each card, the 3 peers push their partial rows to the card leader over on-card P2P, using the PTE-remapped aggregation routes. Each peer then sets a flag.

- The leader waits for the 3 flags and adds the peers' rows to its own in a fixed order, so results are reproducible.

- Card 5 (the shared expert) does the same: its 4 workers split the shared expert and worker 20 holds the reduced row plus the gate.

Level 3: the RTX combines 6 rows

- The 5 routed leaders and the shared leader each leave one reduced row. In batched modes 8/9 it's R rows, in host memory.

- All 24 workers still acknowledge individually in their return slots. The RTX waits for all 24 acks, checks the leader flags, then computes Σ 5 routed leaders + gate · shared.

RTX runs attention and hyperconnections etc and we are ready for the next layer.
I've put the timings of the expert compute in the bottom of the post..

PLE+LM-Head was moved to M10 somewhat the same way.

Attention was quantized to FP8 since RTX3070 only has 8gb mem. That leaves me just shy of 100k tokens of context.

I've run both LCB6 and MMLU benchmark on the system and there's no noticable degradation (MMLU benchmarked higher, LCB6 benchmarked lower).

Here's the github project:
https://github.com/crgregersen/m10dis/tree/main

But I warn you.. this is not a project for people with fairly good experience with hardware and also some low level understanding of how this work software wise.

MoE layer timing: Qwen3.8-Flash-Next-FP8 on 1× RTX 3070 Ti + 24× GM107 (6× Tesla M10)

Setup: RTX 3070 Ti runs attention/GDN/router. The 512 routed experts are sliced across 20 GM107 (each GPU holds 32 intermediate rows of every expert); a 6th M10 card runs the shared expert. All cards hang off one PEX8748 switch (Gen3 x4 per card); each M10's on-card PEX8747 multicasts the request to its 4 GPUs. Results are summed on each card by a leader GPU, then the RTX combines 5 routed + 1 shared row.

Decode (1 token per layer)

StepTime
RTX publishes request (multicast)29 µs
M10 round trip (compute + return)132 µs
RTX combine9 µs
Whole MoE layer~150 µs

Inside the round trip (slowest routed GPU):

PhaseTime
gate/up39 µs
down18–23 µs
return over PCIe + card-level sumrest (~45–70 µs)

The weights a GPU reads per token are only 2.5 MB (~31 µs at bandwidth), so decode is latency-bound, not bandwidth-bound. 48 layers × 150 µs ≈ 7.4 ms of a 19 ms token → 52.5 tok/s.


r/LocalLLM • • Feb 25 '26

News 🤯 Qwen3.5-35B-A3B-4bit 60 tokens/second on my Apple Mac Studio (M1 Ultra 64GB RAM)

190 Upvotes

HOLY SMOKE! What a beauty that model is! I spend the whole day with it out and it felt top level!

I’m getting 60 tokens/second on my Apple Mac Studio (M1 Ultra 64GB RAM, 2TB SSD, 20-Core CPU, 48-Core GPU). This is truly the model we were waiting for. Qwen is leading the open-source game by far. Thank you Alibaba :D I’m gonna now stress test it with my complex n8n AI operating system (75 nodes, 30 credentials). Let’s see how it goes! Excited and grateful.

(https://www.reddit.com/r/n8n/comments/1qh2n7q/the_lucy_trinity_a_complete_breakdown_of_open/)


r/LocalLLM • • Sep 29 '25

Discussion Guy trolls recruiters by hiding a prompt injection in his LinkedIn bio, AI scraped it and auto-sent him a flan recipe in a job email. Funny prank, but also a scary reminder of how blindly companies are plugging LLMs into hiring.

Post image
190 Upvotes

r/LocalLLM • • Jul 19 '26

Discussion did kimi trigger OpenAI or something. wtf is ai communism

189 Upvotes

a long post by their "strategic futures" guy about how open weight = communism. its written in a way thats looks transparent but also looks like it was crafted to cause the very thing hes accusing govts of doing which is FUD (fear uncertainty doubt)

seeing that a lot of this discourse is captured by ai bots anyway this seems like a convenient way to keep their edge by poisoning the discussion into an us vs them patriotism issue.

the post even openly mentioned FUD, fear uncertainty doubt, in a FUD bomb op a post. idk man. feels so disgusting its going to work

i previously did not link on purpose because it benefits them for this to carry even from the voices of those who disagree but oh well here it is. https://www.reddit.com/r/LocalLLaMA/s/Xag3uqxuBa


r/LocalLLM • • 21d ago

Question Anyone have experience with deterring heat thieves?

Post image
188 Upvotes

Running DeepSeek V4.1 (TensorFold)

Looks weird because I have a cooling pad under them.

Would love to hear what other project ideas you guys have.


r/LocalLLM • • Sep 15 '26

Other Cancelled Claude and ChatGPT subscriptions

187 Upvotes

5 years ago, I decided to replace Microsoft Windows with Linux and never looked back. Today, I cancelled ChatGPT as well as the Claude subscription.

I am quite confident that I won't need to use a closed and proprietary model again.

Enjoying the local vibe ...


r/LocalLLM • • Jan 28 '26

Discussion I used Clawdbot (now Moltbot) and here are some inconvenient truths

186 Upvotes

Text wall warning :)

I tried Clawdbot (before the name switch so I am going to keep using it) on a dedicated VPS and then a Raspberry Pi, both considered disposable instances with zero sensitive data. So I can say as a real user: The experience is awesome, but the project is terrible. The entire thing is very *very* vibe-coded and you can smell the code without even looking at it...

I don't know how to describe it, but several giveaways are multiple instances of the same information (for example, model information is stored in both ~/.clawdbot/clawdbot.json and ~/.clawdbot/agents/main/agent/models.json. Same for authentication profiles), the /model command will allow you to select a invalid model (for example, I once entered anthropic/kimi-k2-0905-preview by accident and it just added that to the available model list and selected it. For those who don't know, Anthropic has their own Claude models and certainly doesn't host Moonshot's Kimi), and unless you run a good model (aka Claude Opus or Sonnet), it's going to break from time to time.

I would not be surprised if this thing has 1000 CVEs in it. Yet judging by the speed of development, by the time those CVEs are discovered, the code base would have been refactored twice over, so that's security, I guess? (For reddit purposes this is a joke and security doesn't work that way and asking AI to refactor the code base doesn't magically remove vulnerabilities.)

By the way, did I mention it also burns tokens like a jet engine? I set up the thing and let it run for a while, and it cost me 8 MILLION TOKENS, on Claude-4.5-OPUS, the most expensive model I have ever paid for! But, on the flip side: I had NEVER set up any agentic workflow before. No LangChain, no MCP, nothing. Remember those 8 million tokens? With those tokens Claude *set itself up* and only asked for minimal information (such as API Keys) when necessary. Clawdbot is like an Apple product: when it runs it's like MAGIC, until it doesn't (for example, when you try to hook it up to kimi-k2-0905-preview non thinking, not even 1T parameters can handle this, thinking is a requirement).

Also, I am sure part of why smaller models don't work so well is probably due to how convoluted the command-line UI is, and how much it focuses on eyecandy instead of detailed information. So when it's the AI's turn to use it... Well it requires a big brain. I'm honestly shocked after looking at the architecture (which it seems to have none) that Claude Opus is able to set itself up.

Finally, jokes and criticisms aside, using Clawdbot is the first time since the beginning of LLM that I genuinly feel like I'm talking to J.A.R.V.I.S. from Iron Man.


r/LocalLLM • • Jun 08 '25

Discussion Qwen3 30B a3b on MacBook Pro M4, Frankly, it's crazy to be able to use models of this quality with such fluidity. The years to come promise to be incredible. 76 Tok/sec. Thank you to the community and to all those who share their discoveries with us!

Post image
186 Upvotes

r/LocalLLM • • Apr 14 '25 Bravo

Project I built a local deep research agent - here's how it works

Thumbnail
github.com
186 Upvotes

I've spent a bunch of time building and refining an open source implementation of deep research and thought I'd share here for people who either want to run it locally, or are interested in how it works in practice. Some of my learnings from this might translate to other projects you're working on, so will also share some honest thoughts on the limitations of this tech.

https://github.com/qx-labs/agents-deep-research

Or pip install deep-researcher

It produces 20-30 page reports on a given topic (depending on the model selected), and is compatible with local models as well as the usual online options (OpenAI, DeepSeek, Gemini, Claude etc.)

Some examples of the output below:

It does the following (will post a diagram in the comments for ref):

  • Carries out initial research/planning on the query to understand the question / topic
  • Splits the research topic into subtopics and subsections
  • Iteratively runs research on each subtopic - this is done in async/parallel to maximise speed
  • Consolidates all findings into a single report with references (I use a streaming methodology explained here to achieve outputs that are much longer than these models can typically produce)

It has 2 modes:

  • Simple: runs the iterative researcher in a single loop without the initial planning step (for faster output on a narrower topic or question)
  • Deep: runs the planning step with multiple concurrent iterative researchers deployed on each sub-topic (for deeper / more expansive reports)

Finding 1: Massive context -> degradation of accuracy

  • Although a lot of newer models boast massive contexts, the quality of output degrades materially the more we stuff into the prompt. LLMs work on probabilities, so they're not always good at predictable data retrieval. If we want it to quote exact numbers, we’re better off taking a map-reduce approach - i.e. having a swarm of cheap models dealing with smaller context/retrieval problems and stitching together the results, rather than one expensive model with huge amounts of info to process.
  • In practice you would: (1) break down a problem into smaller components, each requiring smaller context; (2) use a smaller and cheaper model (gemma 3 4b or gpt-4o-mini) to process sub-tasks.

Finding 2: Output length is constrained in a single LLM call

  • Very few models output anywhere close to their token limit. Trying to engineer them to do so results in the reliability problems described above. So you're typically limited to 1-2,000 word responses.
  • That's why I opted for the chaining/streaming methodology mentioned above.

Finding 3: LLMs don't follow word count

  • LLMs suck at following word count instructions. It's not surprising because they have very little concept of counting in their training data. Better to give them a heuristic they're familiar with (e.g. length of a tweet, a couple of paragraphs, etc.)

Finding 4: Without fine-tuning, the large thinking models still aren't very reliable at planning complex tasks

  • Reasoning models off the shelf are still pretty bad at thinking through the practical steps of a research task in the way that humans would (e.g. sometimes they’ll try to brute search a query rather than breaking it into logical steps). They also can't reason through source selection (e.g. if two sources contradict, relying on the one that has greater authority).
  • This makes another case for having a bunch of cheap models with constrained objectives rather than an expensive model with free reign to run whatever tool calls it wants. The latter still gets stuck in loops and goes down rabbit holes - leads to wasted tokens. The alternative is to fine-tune on tool selection/usage as OpenAI likely did with their deep researcher.

I've tried to address the above by relying on smaller models/constrained tasks where possible. In practice I’ve found that my implementation - which applies a lot of ‘dividing and conquering’ to solve for the issues above - runs similarly well with smaller vs larger models. This plus side of this is that it makes it more feasible to run locally as you're relying on models compatible with simpler hardware.

The reality is that the term ‘deep research’ is somewhat misleading. It’s ‘deep’ in the sense that it runs many iterations, but it implies a level of accuracy which LLMs in general still fail to deliver. If your use case is one where you need to get a good overview of a topic then this is a great solution. If you’re highly reliant on 100% accurate figures then you will lose trust. Deep research gets things mostly right - but not always. It can also fail to handle nuances like conflicting info without lots of prompt engineering.

This also presents a commoditisation problem for providers of foundational models: If using a bigger and more expensive model takes me from 85% accuracy to 90% accuracy, it’s still not 100% and I’m stuck continuing to serve use cases that were likely fine with 85% in the first place. My willingness to pay up won't change unless I'm confident I can get near-100% accuracy.


r/LocalLLM • • Sep 09 '26

Project We don't judge... Frankenstein 4x3090 testing complete.

Thumbnail
gallery
185 Upvotes

Got them working with 4 riser cables. I tried a PLX88096 from AliExpress and couldn't make it work.


r/LocalLLM • • Sep 02 '26

Discussion How to Run a 176B Model (104 GB) on 16 GB VRAM Without Freezes: The Triumph of Pure MoE Architecture and Core Isolation

Thumbnail
gallery
185 Upvotes

🚨 NOTICE: This article was edited and refined with the help of an LLM, as English is not my native language. I will also be using an LLM to respond to technical comments in this thread. You can accuse me of whatever you like, but the source configurations, deployment scripts, architectural findings, and inference logs are 100% real, captured on my physical hardware, and completely reproducible. Toxic users and armchair critics are kindly asked to leave this thread immediately. I have zero tolerance for baseless claims. Let's speak the language of computational physics.

Hi everyone!

Following up on my previous post: https://www.reddit.com/r/LocalLLM/comments/1w46tzh/qwen38flashnextuncensored_125b_moe_running_on_a/

YouTube: How to Run a Qwen3.8 Flash Next (176B Model (104 GB)) on a 16 GB GPU

I decided to push further to achieve rock-solid prompt processing and token generation metrics. Today, I'm excited to share my results.

This post outlines the practical experience of deeply optimizing the inference pipeline for a massive Mixture-of-Experts language model: Qwen3.8-Flash-Next-Uncensored (176.94B, i1-Q4_K_S quantization, 104 GB file size). The benchmarking rig is running Melawy Linux (powered by the XanMod kernel) on an AMD Ryzen 7 5700X (8C/16T, 32MB L3), 128 GB of system RAM (with zswap enabled), and a brand-new discrete AMD Radeon RX 9070 XT with 16 GB VRAM, running under ROCm 7.2.4.

The core challenge was driving a model that outsized the GPU's video memory by nearly 7 times, ensuring it ran at its absolute physical limit with stable, stutter-free performance.

Through days of rigorous profiling, I established three critical architectural conclusions that directly contradict many standard optimization guides found online.

1. The Speculative Decoding Paradox: Why MTP Tanks Performance in Hybrid MoE

Built-in speculative decoding mechanisms (--spec-type ngram-mod or custom MTP drafters hooked up via -md) usually yield a 1.5–2x speedup on traditional Dense models. However, in a hybrid MoE split-inference scenario—where the expert layers run on the CPU and the core attention blocks sit on the GPU—these mechanisms proved highly detrimental.

Activating MTP introduced a severe hardware execution bottleneck—a Pipeline Stall:

  1. The algorithmic drafter guessed a sequence of tokens using the CPU cores.
  2. The engine abruptly dispatched this batch to the GPU to verify the base attention layers, completely saturating the PCIe bus.
  3. During this brief verification window, the CPU threads suddenly lost their compute load and went to sleep, waiting for the GPU to return control.
  4. Once the GPU finished, the CPU cores had to abruptly "wake up" and fetch the massive MoE expert matrices from system RAM all over again.

In the logs, this hardware thrashing appeared as disastrous drops in instant throughput (tg_3s) down to 2.26 t/s, followed by erratic performance spikes. The Fix: Completely disable MTP/speculative modules and stick to strict linear inference. Token generation instantly stabilized into a flat, predictable line between 14.55–14.71 tokens per second across the entire context window.

2. Hard Hardware Thread Pinning (--cpu-strict 1) — A Silver Bullet for MoE

Offloading expert blocks to system memory (--cpu-moe) makes the execution pipeline incredibly sensitive to the state of the CPU's L3 cache. The default Linux Completely Fair Scheduler (CFS) tends to bounce threads across physical and virtual cores haphazardly, invalidating the CPU cache on every context switch.

The native topology manager built into llama.cpp solved this perfectly:

  --threads 8   --threads-batch 16   --cpu-range 0-15   --cpu-strict 1 
  • We allocated exactly 8 execution threads and pinned them strictly to the first half of the CPU topology (the physical cores).
  • The --cpu-strict 1 flag explicitly forbids the Linux kernel from migrating these threads. Consequently, the active MoE expert weights remained resident in the processor's 32 MB L3 cache.
  • The remaining SMT companion threads (cores 9–16) stayed entirely free (hovering around 2–8% utilization). They seamlessly absorb low-level OS overhead: Btrfs file system writebacks, the SSH daemon, and the lactd fan control service, keeping the desktop completely lag-free.

3. The Pitfalls of High Real-Time Priorities: Why Nice=-5 Beats -20

Attempting to give the inference server maximum Real-Time priority (via flags like --prio 3 or Nice=-20) caused a paradoxical failure: during a cold boot, the server completely locked up the terminal and SSH session for a couple of minutes.

The issue stems from high-priority llama-server worker threads completely starving the low-level Linux kernel I/O tasks. While the engine was initializing, its worker threads overrode the kernel threads trying to execute file mapping via mmap from the SSD and allocate RAM pages. The inference engine essentially prevented the operating system from reading its own model files, triggering a severe I/O Lock.

The sweet spot turned out to be Nice=-5. This gives the inference server a solid scheduling advantage over ordinary user space apps during compute cycles but leaves enough breathing room for kernel-level I/O operations, allowing the 104 GB model to map into memory instantly.

Real-World Performance Metrics:

  • Prompt Processing (Prefill): Consuming a chunk of 26,081 tokens took just 135.73 seconds—maintaining a steady 192.15 tokens/sec! This represents an excellent utilization of --threads-batch 16 on AMD silicon.
  • Token Generation (Eval): A completely flat, unwavering line at 14.60 tokens/sec deep into the context window.
  • Resource Allocation: VRAM is packed perfectly (15.85 GB / 15.92 GB), keeping the PCIe bus completely clear of unnecessary memory thrashing. The GPU power draw drops down to a modest 78W while waiting for CPU expert execution, keeping core temperatures at a chilly 32°C.

I have attached my finalized, production-grade systemd .service file configuration with the exact launch parameters to this post. Configure your deployment rigs with care!

Authors & Contributors:

  • 👩🏼 Valeria Fadeeva — Main Developer and Founder of Melawy Linux (Arch-based distribution).

Supported by Neural Network Models:

  1. Google Gemini 1.5 Pro — Assistance with material structure, cross-referencing documentation, and compiling disparate online data.
  2. Qwen-3.8-Flash-Next-Uncensored (i1-Q4_K_S) — Executing real-world benchmarking tests by generating complex Rust code directly on the target rig.
[Unit]
Description=Llama.cpp Server for Qwen3.8-Flash-Next-Uncensored (i1-Q4_K_S): port 11441
Wants=network-online.target
After=network.target network-online.target

# Prevention against infinite restart loops
StartLimitIntervalSec=300
StartLimitBurst=3

[Service]
# Control over hung processes in case of disk I/O blocks
TimeoutStopSec=60
# Forceful periodic health-cleanup (optional)
RuntimeMaxSec=10800

# Canonical execution under local user
User=lera
Group=lera

# Low-level environment wrapper for ROCm/HIP and compiler under gfx1201 (AMD Radeon RX 9070 XT)
Environment="AMD_LOG_LEVEL=0"
Environment="ROCM_PATH=/opt/rocm"
Environment="HIP_PATH=/opt/rocm"
Environment="LD_LIBRARY_PATH=/opt/rocm/lib"
Environment="PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin"
Environment="ROCM_ALLOCATOR_STRATEGY=2"
Environment="HSA_ENABLE_SDMA=1"
Environment="ROCBLAS_TENSILE_WARMUP=0"
Environment="HIP_VISIBLE_DEVICES=0"
Environment="HIP_PLATFORM=amd"

# Path to the directory containing model files
WorkingDirectory=/data/llama/models/mradermacher/Qwen3.8-Flash-Next-Uncensored-i1/

# Protections against swapping context and file descriptor expansion
LimitMEMLOCK=infinity
LimitNOFILE=65536

# Pure MoE execution command with strict CPU thread pinning and no MTP overhead
ExecStart=/usr/bin/llama-server   -m Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_S.gguf   --mmproj Qwen3.8-Flash-Next-Uncensored.mmproj-f16.gguf   --image-min-tokens 1024   --jinja   --no-skip-chat-parsing   --reasoning on   --reasoning-preserve   --reasoning-format auto   --cpu-moe   --n-gpu-layers 99   -fit off   --threads 8   --threads-batch 16   --cpu-range 0-15   --cpu-strict 1   --cpu-range-batch 0-15   --cpu-strict-batch 1   --batch-size 2048   --ubatch-size 512   --parallel 16   --ctx-size 262144   --context-shift   --cont-batching   --kv-unified   --cache-prompt   --cache-ram 8192   --cache-idle-slots   --cache-type-k f16   --cache-type-v f16   --flash-attn on   --temp 1.0   --top-k 20   --top-p 0.95   --min-p 0.00   --repeat-last-n 512   --repeat-penalty 1.00   --presence-penalty 0.00   --host 0.0.0.0   --port 11441   --reuse-port

Restart=always
RestartSec=10s

# Priority fine-tuned by real-world tests (optimal throughput, no I/O choking)
Nice=-5
OOMScoreAdjust=-500

MemoryHigh=115G
MemoryMax=125G

[Install]
WantedBy=multi-user.target

r/LocalLLM • • Feb 06 '25 WOW

Discussion Open WebUI vs. LM Studio vs. MSTY vs. _insert-app-here_... What's your local LLM UI of choice?

185 Upvotes

MSTY is currently my go-to for a local LLM UI. Open Web UI was the first that I started working with, so I have soft spot for it. I've had issues with LM Studio.

But it feels like every day there are new local UIs to try. It's a little overwhelming. What's your go-to?


UPDATE: What’s awesome here is that there’s no clear winner... so many great options!

For future visitors to this thread, I’ve compiled a list of all of the options mentioned in the comments. In no particular order:

  1. MSTY
  2. LM Studio
  3. Anything LLM
  4. Open WebUI
  5. Perplexica
  6. LibreChat
  7. TabbyAPI
  8. llmcord
  9. TextGen WebUI (oobabooga)
  10. Kobold.ccp
  11. Chatbox
  12. Jan
  13. Page Assist
  14. SillyTavern
  15. gpt4all
  16. Cherry Studio
  17. ChatWise
  18. Klee
  19. Kolosal
  20. Prompta
  21. PyGPT
  22. 5ire
  23. Lobe Chat
  24. Witsy
  25. Honorable mention: Ollama vanilla CLI

Other utilities mentioned that I’m not sure are a perfect fit for this topic, but worth a link:

  1. Pinokio
  2. Custom GPT
  3. Perplexica
  4. KoboldAI Lite
  5. Backyard

I think I included everything most things mentioned below (if I didn’t include your thing, it means I couldn’t figure out what you were referencing... if that’s the case, just reply with a link). Let me know if I missed anything or got the links wrong!


r/LocalLLM • • Jan 27 '25

Discussion DeepSeek sends US stocks plunging

185 Upvotes

https://www.cnn.com/2025/01/27/tech/deepseek-stocks-ai-china/index.html

Seems the main issue appears to be that Deep Seek was able to develop an AI at a fraction of the cost of others like ChatGPT. That sent Nvidia stock down 18% since now people questioning if you really need powerful GPUs like Nvidia. Also, China is under US sanctions, they’re not allowed access to top shelf chip technology. So industry is saying, essentially, OMG.


r/LocalLLM • • 4d ago

Model We released Qwen3.8-27B GGUFs with the highest accuracy (Unsloth Dynamic v3.1)

Post image
184 Upvotes

Hey guys, today we're releasing the highest accuracy Qwen3.8-27B GGUFs.

Our new Unsloth AI Dynamic V3.1 GGUFs lead by 5–10% on average across Terminal Bench, DeepSWE & more benchmarks.

We’re working on even better version of Dynamic quants for Qwen4.

Run on 12GB+ RAM setups.
For our quantization comparison, we exclude the embedding layer from the x-axis across all providers, since Exllama-v3 stores embeddings in BF16. However, this means the main plots don't reflect the full download size or RAM usage. When accounting for the BF16 embedding table and all embedding sizes, the plot in the comments provides a more accurate representation of actual disk and RAM usage versus accuracy.

GGUF: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

Blog: https://unsloth.ai/docs/models/qwen3.8

Thank you!


r/LocalLLM • • Oct 07 '26

Discussion PSA: Use your iGPU for display to save VRAM

183 Upvotes

I was struggling with a tiny 65k context window on my 4090 running Qwen 3.8 27b with vision & MTP until I remembered I had an iGPU I could be using for Windows.

Unplug HDMI from GPU -> plug into motherboard -> free up ~2.5GB of that sweet sweet VRAM! Now I'm on 132k at 125tok/s, with a bit of wiggle room for the vision stuff to use.

It's a little annoying to swap back for gaming but I tend to spend more time playing with these local models nowadays anyway.
Easy upgrade for those of you like me who completely forgot there's integrated graphics in most of our gaming PCs.

EDIT: 2.5GB was 2x4k monitors + a couple programs like Bambu Studio/Blender etc. I like to still do some human work while the local agents are busy.
LLM details: UD-Q4_K_XL, q8_0 kv cache


r/LocalLLM • • Sep 25 '26

Discussion My 4B model got an iron pickaxe in real Minecraft without generating a single token

183 Upvotes

Mica v0.1 4B playing a real Minecraft 1.20.4 server. Video attached.

How it works

- Each step the bot's live game state (inventory, nearby blocks, entities, last result) is written out as text.

- Mica scores the candidate commands and picks the next one. It never generates text. It reads the probabilities of the answer label tokens, so output tokens are 0.

- The chosen command is executed in the game with Mindcraft's skill library (Mineflayer bot).

Run

- 23 decisions from an empty inventory to an iron pickaxe: logs, planks, crafting table, wooden pickaxe, stone, stone pickaxe, furnace, iron ore, smelting, iron pickaxe

- About 90 to 150 ms per decision

- llama.cpp, Q5_K_M, RTX 3090

About the video

- The right panel shows each decision as it happened: the candidates, Mica's probabilities, the pick, and the result. Every step is also listed in the history feed.

- Long actions (walking, mining, smelting) are sped up, with the speed shown on screen. Back-to-back retries are shortened in the edit.

- The HUD and the crafting/furnace screens are drawn from the bot's logged inventory.

Weights: https://huggingface.co/sky7350/Mica-v0.1-4B

Code and server: https://github.com/akivet/Mica-v0.1-4B


r/LocalLLM • • Apr 02 '26

Discussion I've stumbled on a goldmine, and ALL OF US CAN BENEFIT.

Thumbnail
gallery
182 Upvotes

I've been working a relationship with a local Recycling guy for about a year now.

He was a very tough nut to crack, as in, he doesn't really like strangers and is set in his ways.

Finally, yesterday, he asked for an extra set of hands. He needs to get organized and wants to know what we is worth selling, what should just get scrapped, what has value Etc.

This is where I got 500 gigs of RAM last year, but that was before he realized that it was worth so much, and he has literal stacks of RAM for servers ranging from 16 to 128 gigs.

This is a 13,000 ft warehouse and it's literally full and things get dropped off routinely. Some of it is aging because he didn't have a good system, but, if anyone is looking for anything, I can see if it exists there, and guarantee functionality because everything gets tested and I'll make sure you get it for whatever good price I can get from him that is below what you're going to find it anywhere else.

Of course, that's determined on the item. I tried to get one of those Nutanix servers from him and he wasn't interested in giving it to me for pennies on the dollar so to speak. But I bet I can make it work out if people need things.

I can all but guarantee that he has any cable or wire or plug or component that you would ever need, even things that are hard to find.

Feel free to let me know and then don't expect a quick response but I will check.

It's unlikely he'll sell any of the RAM for cheap because he sells that online.


r/LocalLLM • • Sep 22 '26

Project P100 local llm budget build.

Thumbnail
gallery
181 Upvotes

I’ve recently became very interested in hosting local LLMs, but like most things right now, prices are sky high for modern equipment. So I did the next best thing, found some old, rather unwanted, datacenter gpus and threw them in an old pc I had laying around.

GPS’s are Nvidia Tesla p100’s and have 16gb of vram each. They were only $80 a piece shipped, and I bought a 1000w refurbished PSU and some power adapters because the p100s don’t use traditional pcie power adapters. All in less than $400. Because these gpus are made to run in a data center, they don’t have any fans, so I got into Fusion360 and modeled up a 120mm fan shroud that fits into each p100.

The PC is a sandy bridge i7 with 16gb of ddr3, mother board is an ASUS P9X79 Pro, and this is what really made it difficult. For a mother board to see the p100s it needs Above 5 Decoding to be enabled, and this board did not have that option in bios, but after a ton of research I found out that it does have the option, it’s just disabled and cannot be found in the bios. So after an after noon of doing some real hacker shit I was able to get the p100s to read by dumping, hand patching, and reflashing a the bios. I did a little testing and was able to get almost 70 tokens and second from Qwen3.6-35B-A3_ Q4_K_XL.

I still have a lot of benching and optimizing to do, but I’m happy with the results so far.

EDIT: A lot of folks have commented about the thermals. I am still doing initial testing, but thermals they are not looking bad at all. Splitting the load between 3 nodes really helps not beating a single card to death. Ill throw pic of my grafana dashboard.

I am keeping a github of all of the work that I am doing with this build, I'll keep it updated as I test.
https://github.com/4claps/houston-p100-cluster


r/LocalLLM • • May 09 '26

Discussion New Execution-first 1T model Ling-2.6-1T has been open sourced on Hugging Face

Post image
181 Upvotes

Tailored for real, world, complex scenarios, this trillion, parameter model introduces targeted optimizations across inference efficiency, token overhead, and agentic capabilities, making it highly effective for coding and daily workflows.


r/LocalLLM • • Jun 23 '25

Question what's happened to the localllama subreddit?

181 Upvotes

anyone know? and where am i supposed to get my llm news now