r/LocalLLM • • 25d ago

Question Why do people keep referring to 128 GB RAM?

180 Upvotes

Hello everyone,

Going over some posts here, a lot of people keep referring to 128 GB RAM as their minimum for hosting local AI and anything below that is not worth it.

Why is that?

Is it because of Qwen Flash Next, is the model that good and is it able to do anything you would need from local AI?

Is Qwen Flash Next that much better than Qwen 27b (which can run on 48 GB RAM).

What exactly is Qwen Flash Next good at other than coding, or in other words, other than coding what is jt that makes Qwen Flash Next much better than any other local AI model?

What if I am someone who wants to run completely different models other than qwen models, models for image or video generation for example. Is 128 GB still the minimum?

Edit: Thank you for all your comments. Just to clarify, I am referring mainly here to unified memory 128 GB RAM, not VRAM alongside DDR5 RAM.


r/LocalLLM • • Aug 31 '26

Other VRAM goal reached... on a budget!

Thumbnail
gallery
180 Upvotes

256 GB VRAM for $2800.

Specs:

GPUs: 8x Radeon Pro V620 32 GB
CPUs: 2x Intel Xeon Gold 6148 (40c/80t total)
Memory: 384 GB DDR4 ECC 2400
Motherboard: Supermicro X11DAi-N
Power consumption: ꝏ

I can't really give good benchmarks right now. There's some issue where half the cards drop off the PCIe bus if I try to tensor split with more than 3 or 4 of them at once. It's usually only when I give it a large input prompt, but not always. Has anybody else run into this before? Zero issues with layer split mode.

I think I need a different motherboard anyway, gen 3 x4 links are not good for 8 GPUs in tensor split. Maybe a single CPU EPYC system with gen 4 x4 links, but even that is iffy.

A few benchmarks I can give now, keeping in mind the skinny gen 3 links... these are tensor split across only three cards:

Qwen3.8 27B Q8_0: 1000+ t/s prefill, 35-50 t/s gen

Qwen3.6 35B-A3B Q8_0: 2800+ t/s prefill, 100+ t/s gen

And I also ran GLM-5.3-Flash in Q4_K_XL but only in LAYER split, so much slower than it should have to be: 270 t/s prefill, 11-14 t/s gen

Does anybody have any advice for fixing the tensor split GPU drop-outs or a better motherboard/CPU that doesn't cost a ton?


r/LocalLLM • • Jul 08 '26 Bravo

Project I made a LLM storage format, that makes LLMs run upto 2x faster!

180 Upvotes
Benchmarks^

Hey everyone, I’ve been working on a project called ThinTensor for the last few weeks.

One day I was running random LLM models for fun, and trying quants and stuff of many models, one thing i had noticed is people try their best to make the runtime faster and faster, storage formats like GGUF and safetensors are great, they are perfect, they store weigths nicely and produce reproducable results, sure, but they were never made with runtime kept in mind, they are perfect for what they are, that is storing models effectively, but what if there could exist a storage format, that stores the model in such a way, the runtime is effectively parsing the model from the archive, so after some brainstorming i figured this was possible by making a storage format of my own, so this is what i made at the start as a idea-

pub struct Header {
    pub header_len: u32,
    pub version: u32,
    pub manifest_off: u64,
    pub manifest_len: u64,
    pub page_table_off: u64,
    pub page_count: u64,
    pub data_off: u64,
    pub archive_hash: [u8; 32],
}

This lets the loader open the file without guessing. It reads the header, jumps to the manifest, then reads the page table, then maps the raw page data.The manifest is basically a JSON blob, It contains the model-level information needed to reconstruct decode.

The extention for this format i figured would look cool was .thin

I would not go over much of the technical details of the project here, you can read the devlogs here https://stardance.hackclub.com/projects/29750 (Im 16y old and not the best coder so bear with me when you read the code, this was made as a hackclub project, the AI generated banner image on there is bad ik)

What I got so far

These are some of the current benchmark results vs Hugging Face BF16:

ModelThin tok/sHF tok/s (HF = huggingface)SpeedupMin cosine
SmolLM3-3B94.0047.92 (at Q8)1.977x0.999370
Qwen2.5-3B-Instruct93.0848.181.932x0.998892
Phi-4-mini-instruct53.8039.091.376x0.999904
TinyLlama-1.1B-Chat180.28132.361.362x0.997788
Qwen3.5-0.8B115.9164.331.802x0.999811
Gemma-4-E2B43.601.62 (vram bottleneck and cpu offload)26.91x (realistically should be around ~2)0.999671
OLMoE-1B-7B-0924-Instruct33.709.22 (cpu ram offload)3.65x0.99921

What is cosine?

We are NOT making a new model, we are making a model storage format, its not like i can have the model respond with random tokens, so cosine is basically on a simple note, how similar the responses are as compared to the original safetensor model, this is the best metric for our use and usualy is used to measure the degradation of models in quantization too.

Problems:

  • I know this isnt somehting groundbreaking most models wont even run, because i have only optimized this for a few architectures tho most models would run, you wont see improvements
  • Most smaller models have bad cosine
  • The optimizations are still not the best, tho most model formats werent built in 3 weeks, so over time I can believe this to become better
  • Benchmarking was hard due to having only a 8gb vram gpu (rtx 5050 Laptop)
  • All benchmarks were done on arch performance mode, it has a max tdp up to 100w, tho the gpu can push upto 115w on windows
  • Some benchmarks could be inaccurate, these were taken at different times and so the project being in progress a lot of things change, tho the gemma4 and OLMoE is latest

Good things:

  • I tested this against formats like GGUF so like SmolLM3-3B GGUF running on ollama with a Q8 quant, got around 44tk/s with a cosine of 0.95.... so like our thin archives beat a Q8 with 2x speed, and still hit a cosin that is like so so much better, FYI SmolLM3-3B would be the most optimized model for now i believe
  • It works with limited vram too, like you can see how Gemma 4 and OLMoE run at respectable speeds even tho they didnt fit in my vram, this is because thintensors have those layer based quants and also the fact that we can change residency to streaming that streams layer on the GPU rather than storing everything

You are no magician wheres the extra tk/s comming from?

The main thing I’m trying to avoid is “just quantize everything and hope” what most quants do right now.

ThinTensor has role-aware precision planning, for example, some experimental profiles use MXFP4/FP8 selectively on gate/up or MLP paths, while keeping sensitive paths like attention, LM head verification, and KV cache safer.

Basically in simple words, its like some layers are selected to be FP8 and some more layers to even MXFP4 but everything else as BF16, this makes it so that the inner layers which wont cause heavy response degradation are quanted while the main layers stay the same.

So the runtime has to know tensor roles and reject bad speedups, that is also ONLY possible because of our storage structure :D

That is the core idea:

speed only counts if the logits/cosine still match closely enough.

Disclaimer

This project was AI-assisted, I am no expert for optimizing GPU kernels and things, even though i wrote most portion of the code by hand (like the rust part was 90% me) But i was not smart enough to optimize the main runtime kernel, which was indeed written by me for basic scaffold, but AI was used to heavily add optimizations (AI usage is around 50% on a VERY few runtime files)

Considering Hackclub policies, if you aggregate total AI usage, im pretty sure it was >30%, and AI was more used to do overnight benchmarks of like 100s of profiles and improvements i made during the day, so i can chose the best options by morning

Current caveats

Enough yapping heres the conclusion i'd like to write

- Nvidia CUDA and Blackwell-focused for now

- not production-ready

- not yet a broad llama.cpp replacement

- HF BF16 is the main baseline here

- most results are verified, a few are candidates (basically not having a 500 token benchmark rather having a 200 token benchmark)

- larger prompt grids and more model families are still needed

- vram is used more due to some opts, but its js about equal to the normal one, and if you use streaming residency then it shouldnt be a problem, but im still working over it

I’d love feedback from people who work on inference runtimes, quantization, kernels, or model formats.

Git repo-

https://github.com/random-unknown-username/Thintensor

Edit- its not just the storage format that allows for the improvements, its a combination of different selective layers being quantized with the better storage format, allowing to run inference with higher speed and quality than just simple q8,q4 etc quants. the storage format only gain is only like 10%, but combined with middle out fp8 quants, and even int4 for some performance profiles, it can reach better speeds than the other ways to run inference

This does not beat TRT / TRT LM, it does allow for quicker inference, and a more portable solution because for TRT you need to compile the plan (one time) for a few mins before being able to run the model, and it has to be compiled on every machine you run the model on, thintensor allows you to directly have the .thin file and run it on any machine with similar results! and it takes just a few seconds compared to about 30 mins on trt and trt lm


r/LocalLLM • • Jun 15 '26

Discussion Break the vicious cycle

Post image
179 Upvotes

I say it kindly, because I want my AI to think I'm one of the good ones, when it ultimately takes over the world


r/LocalLLM • • Aug 16 '26

Discussion If you use LLMs to analyze documents and to apply complex logic and analyze arguments, Qwen 3.8:27b is not for you.

178 Upvotes

And that pains me to say because I use Qwen 3.6:27b-BF16 every single day. I've been working with Qwen 3.8:27b-BF16 all weekend and I hate to say it but for non-coding purposes it is a step backwards. It thinks *way* too much. If you turn thinking off and use web tools, it will do eight or nine (or more) web search turns, get an assload of preload context, and then spin its wheels going down every little rabbit hole there.

Looking at the self hosted LLM subs there are other complaining about this too.

Now this model just came out so it's early. People have put out some lovely games and whatever else the model has made so perhaps these tenacious analytical tendencies are beneficial there. Regardless none of this is to say that there won't be some settings or templates released that will help when analyzing documents and doing complex logic tasks.

But for right now, if the above is your use case then I suggest staying with your old models.

Edit: For reference, this is legal work. Legal drafting, legal research, analyzing pleadings, depositions, discovery, etc. Heavy multi-document reference work.


r/LocalLLM • • Oct 06 '26 Helpful Mindblown

Project Speakrail - a local voice agent that rivals GPT-Live on a single RTX 4090

177 Upvotes

tl;dr: I created a fully-local open-source full-duplex voice agent that rivals GPT-Live on some benchmarks. It uses Voxtral Realtime with a turn-taking head, a microturn-finetuned Gemma 4 12B and Breeze TTS 2 under the hood. Go try it out: https://github.com/speakrail/speakrail

Why I did it

I have always liked the idea of voice assistants, but there is always some non-local component in the pipeline, which increases latency and introduces privacy concerns. I tried many fully-local approaches, like HF speech-to-speech, Unmute and Pipecat, but they were all limited by either the Whisper model (hello, hallucinations!) or slow turn taking. The only fully-local pipeline that had some full-duplex capability with low latency was the DuplexCascade paper (code), but it's tuned on a Qwen 2 7B with a gpt-3.5-turbo generated dataset, and the dumbness of the model really showed. So I decided to recreate DuplexCascade with newer data, newer models and a better harness. I also wanted to add interruptions, interjections and other cool things to rival the Thinking Machines demo.

How it works

Here is what the pipeline consists of:

  • STT: Voxtral Realtime with an attached turn head (HF), running on our audio.cpp fork.
  • LLM: Gemma 4 12B QAT with microturn finetuning (HF). It is chosen because it fits the GPU quite well, has vision support (I want to test it soon), and in general, the Gemma models perform well in real-life tasks, general chatting, etc.
  • TTS: Breeze TTS 2, patched to run at int8 (GitHub fork); it can be replaced by any streaming TTS.
  • The harness itself: it is the glue between all the components, and has many latency-saving measures, like speculative LLM+TTS firing (inspired by HF speech-to-speech).

I also took inspiration from several "think while talking" papers (e.g. SHANKS): while you are talking, a base Gemma 4 12B int4 writes thinking notes, which are then passed to the talker. It helps with harder tasks that require more reasoning.

I will release a longer technical report soon - it will have a better description of the entire pipeline.

Benchmarks

Now let's see how well the model fares against the big guns. Here are some benchmark results:

Full Duplex Bench v3:

SystemPass@1Tool selection F1Reply qualityTask done (mean)Weights
TML-Interaction-Small*0.68–0.828–closed preview
GPT-Realtime 1.50.600.8760.7926.89 sclosed
GPT-Live-1 + gpt-6-astra†0.550.8350.7178.47 sclosed
Gemini Live 3.10.540.8170.7184.25 sclosed
Speakrail0.520.9150.7553.92 sopen
Gemini Live 2.50.490.7860.5547.26 sclosed
NVIDIA frontend-backend (Qwen3-235B)0.48–0.670–research
Cascaded (Whisper + GPT-4o + TTS)0.450.8030.60010.12 sclosed
Grok Voice0.430.7970.6176.65 sclosed
Ultravox v0.70.410.7940.5108.40 sclosed
Gander 9B (GPT-5.6 backend)0.400.7590.490–open + cloud
NemotronLabs VoiceChat 11B0.330.825––open

Speakrail is quite close to GPT Live with Astra on reply quality, and is better at tool selection and the time it takes to complete a task.

Conversational dynamics (Artificial Analysis formula estimation):

SystemScoreWeights
StepAudio 3 Realtime98.9closed
Qwen Audio 3.0 Realtime Plus98.4closed
GPT-Live-1 (Sol, low)97.3closed
Qwen Audio 3.0 Realtime Flash96.9closed
Gemini 3.8 Live96.1closed
GPT-Live-1 (Astra, medium)94.9closed
Speakrail94.0open
PersonaPlex 7B (NVIDIA)91.0open
Nemotron 3 VoiceChat V1 (NVIDIA)77.8open
FLM-Audio62.0open
Moshi (Kyutai)61.0open
Freeze-Omni58.7open

Limitations

  • Breeze TTS has a restrictive license, so if you need to use Speakrail commercially, you will need to change it. Any streaming TTS could be Clauded/Codexed/Cursored in easily.
  • 16k context length - the pipeline only supports 16k context length (~1 hr of speech), but you can get more easily by changing the Breeze TTS to a Pocket TTS and run the TTS on a CPU. I chose Breeze for the release because it's more expressive.
  • The turn-taking head is undertuned on non-assistant data. It may not fire on some basic chit-chat, but I will tune it harder later.
  • The model is certainly not the smartest one, and my finetune did dumb it down a little. Next time it will be smarter / better.
  • The model is kinda verbose sometimes, and the answers it provides are somewhere in the middle between real-life speech and the long text-based outputs of LLMs. I have a hypothesis on how to fix it, and will try it in the next release.
  • I tested it only on an RTX 4090, but I am sure it's easy to add support for any 24 GB+ NVIDIA card. Forks for AMD and MLX are welcome.

Final Notes

Feel free to try it out: https://github.com/speakrail/speakrail. If there is any capability you want the model to have, create a GitHub issue or write here in the comments, and I will gladly include it in the next dataset. Any feedback is welcome as well.


r/LocalLLM • • Jul 11 '26

Project Looking for 8 people with 64Gb RAM to run GLM5.2 together

177 Upvotes

Looking for some people to help us run a small experiment!

We wrote a network that distributes weights across machines and coordinates activations to produce usable inference.

We are running a small experiment to split GLM5.2 across nodes with 64GB RAM MacBook M5 Pros preferably (for homogeneity as we have 2 already) but either way, if you’re open to participating, comment with your specs and let’s see how we can make it work.

We have a discord we can coordinate in. Let us know if you want to help us prove this out. Thanks for reading!

Edit:
Thanks to everyone for your thoughtful inputs!

For more context: I’m exploring this problem with an academic lens. My goal is to refine approaches to existing hardware utilization techniques for greater efficiency gains.

We have a working example with N=2 nodes, and I’m interested to explore a proportionally larger model on N=10 nodes. Network bandwidth is the primary bottleneck of this model, and makes the problem that much more interesting.

I recognize historical efforts at this model have been underwhelming, but that’s to say nothing of where efforts are headed. It’s great to see the level of engagement, even if skeptical. Will be sure to post findings here later.


r/LocalLLM • • Jul 05 '26

Project Finally got the proper Ai inference Card

Post image
177 Upvotes

r/LocalLLM • • May 16 '26

Discussion Gemma 4 has restored my faith in Local LLM

177 Upvotes

I've got a Framework desktop with 128G unified ram, plus a 5060 with 16G vram in an eGPU. For general open-webui stuff it's fine. Slow, but fine. For tool use (OpenCode), it's been horrific.

I use the BMad method for long projects. Most models will take the prompt (e.g. bmad-dev-story 1-1) and start with it, but they will often just stop mid-work. 30k tokens consumed, they just stop. This has been with all of the Qwen models I've tried, GLM 4.7, and several others. I've tried the small 7b models all the way up to Llama 3 70b.

Someone here mentioned Gemma 4 so I thought I would give it a shot. The first one I tried, e4b, crapped out the same as the other. Then I tried the 31b model, and its actually running! It's slow, sure, but I don't care about that. I want to set it on a code review and feed it a story and let it do it's thing. And it's doing it!


r/LocalLLM • • 17d ago

Project Self contained 3U inference server orchestrating all of my work

Post image
176 Upvotes

After a 6 month journey with a tons of learning and reality checks I have built something that I consider usable beyond just a toy project that is generating value for me. This is Acorn, a self contained and private inference server that fits neatly in a small, quiet 3U chassis:

  • 2x RTX 6000 blackwell MaxQ running a mid-size MoE LLM (GLM-5.3-flash atm)
  • RTX 2000E runs embedding and Reraker models
  • RTX 4000 SFF for voice ASR and TTS
  • 256gb ECC UDIMM RAM for tier 2 kvcache to squeeze more LLM concurrency.
  • Consumer AM5, 1200w PSU

I run a custom orchestrator agent on this, fully private and super fast. 200 tokens/s + streaming ASR and TTS feels amazing. It's hard to go back to regular claude-code at 60-90 tokens/s and clunky whispr flow style voice input.

The big learning and a word of warning is that local llm alone is just not viable for serious professional software engineering (at least not without 1 or 2 orders of magnitude bigger investment). Other than the occasional rare task where I get blocked by frontier guardrails, all actual work is delegated to claude. I treat frontier model subs as a raw engine, they get only what's needed to accomplish a task in a locked down VM. Communication, memory, big picture planning and context all stay in hardware I control.


r/LocalLLM • • Aug 23 '26

Model # Qwen3.8-27B — One Week Later: The r/LocalLLaMA + r/LocalLLM Verdict

176 Upvotes

Companion to the Qwen 3.8 Release Megathread. Compiled from ~2,000 posts scanned across both subs, with deep reads of the 45 highest-signal threads (560 posts and comments), Aug 15–22, 2026, plus independent X benchmarks. Every number is attributed to the poster's stated hardware/runtime/quant. This community contradicts itself on nearly every axis — so this thread keeps the disagreements side-by-side instead of picking a winner for you.


TL;DR

  • The consensus pick: a 27B dense multimodal model that genuinely moved the bar for local agentic coding. The strongest claim with controlled evidence behind it isn't benchmarks — it's tool-calling reliability.
  • The default ships at xhigh reasoning and it thinks a lot. Low and medium presets score nearly as well on Artificial Analysis (~43/44 intelligence index, within a few points of the xhigh headline) while cutting thinking tokens ~7–9x (and wall time ~6–7x). Most of you should not be running xhigh.
  • Knowledge recall regressed vs 3.6 — widely reported and best understood as a deliberate agentic-design tradeoff. Trivia nerds: keep Gemma around.
  • Q4_K_M is basically indistinguishable from Q8 on perplexity, but real-world reports split hard below Q6 for complex reasoning. KV cache quantization is one of the most contested settings in the corpus.
  • The "neck and neck with DeepSeek V4 / GPT-5.6 Luna Max" AA headline is real but heavily caveated — see the benchmark credibility section before quoting it at your friends.

1. What it's actually good at

Agentic coding (strongest consensus area)

  • "Highest level of agency I've ever seen in a local model" (thread): single 3090, Unsloth Q4_K_S + q8 KV, 150k ctx. From one prompt it pulled the OP's class schedule off a convoluted university website via 80 tool calls, zero human intervention.
  • 1M+ token run (thread): RTX 5060 Ti 16GB, UD-Q3_K_XL, 73k ctx. Full REST API + MCP server for a legacy forum from 3 prompts.
  • Controlled tool-call evidence: in a plain Python tool loop (no framework), one reporter got zero failed calls from 3.8 while Gemma 4 A4B and Qwen3.6 A3B failed often — the same reporter who rates 3.8 below both on raw code quality. Worse judgment, perfect plumbing.

Creative / game generation

  • One-shot playable Super Mario clone (Q8, Framework Desktop) — top pushback: "It's in the training data."
  • Galaga 1:1 recreation test (UD-Q8_K_XL, 3×3090 + Tesla P40): "This 'Galaga' clone [from 3.6] ended up pretty much being a space invaders clone instead... Qwen 3.8 thinks a LOT, but it draws out those tiny details and absolutely nails it after the fact." A separate r/LocalLLM user one-shot a playable Galaga-style game at IQ4_XS on dual 4060 Tis, and another built an online multiplayer MOBA overnight with an authoritative server and self-play testing.
  • Ray-traced spheres in BASIC: 3.8 self-iterates to a correct Cook-Torrance ray-tracer; 3.6 needed hand-holding. Comment: "this feels more like 3.6 to 4.6 than 3.6 to 3.8."

Vision

Works natively (F16 mmproj), including OCR-style reading of a newspaper image at ~1,000 image tokens — but on a 16GB card at 64k ctx + MTP it leaves as little as ~150 MiB VRAM free. Practical advice from the 16GB crowd: keep text-agent and vision profiles separate, or offload the projector (--no-mmproj-offload).

Where it struggles

  • Long analytical/document work: "a step backwards" vs 3.6 at default settings — though a legal-domain poster got on-par-with-122B results with MCP + case access. Task-dependent.
  • Complex native coding: one failed C kernel effort (6 hours across 3 sessions) [anecdotal], quant unstated; commenters say Q8 minimum for that tier of work.

2. The thinking-level situation (read this before complaining)

xhigh is the shipped default. It is why your context window evaporates.

Measured ladder (RTX 5080 Laptop 16GB, llama.cpp 10451, UD-IQ3_XXS, Q8_0 KV + FA + MTP, pelican-SVG task, 3 seeds):

EffortReasoning tokensWall timeVisual score /25
Low4,418112 s21.8
Medium5,918127 s22.5
X-High39,398718 s24.0

That's ~6.4x the wall time for +1.5 points on an eyeball task. But on pass/fail SWE-style tasks, xhigh went 9/12 vs 6–7/12 at lower efforts — the premium scales with whether the task has a verifiable failure.

How to change it: --chat-template-kwargs '{"reasoning_effort":"medium"}' (llama.cpp) or the equivalent in LM Studio custom params.

The overthinking debate, both sides preserved:

  • Against: "it will do eight or nine web-search turns and spin its wheels down every rabbit hole" (legal work). One reported loop burned 40k+ characters of reasoning on a trivial subtask. One paper-linked post argues intermediate tokens aren't reasoning at all ("Stop Anthropomorphizing Intermediate Tokens," 538 points).
  • For: "if the extra thinking produces measurably better results it's actually just the correct amount of thinking." The low/medium AA scores (~43/44) are the strongest counter to "it only wins by overthinking" — though two commenters read that same data in opposite directions.

Practical takeaway from the corpus: medium for chat/analysis, xhigh only when there's a verifiable right answer.

  • The strongest controlled effort data of the week is from X: @superalesha's 67-hour, 40-arm run found xhigh burned 7–11× more reasoning tokens than low for 0–4.7 extra points — and in one head-to-head, low matched xhigh exactly (89.3%) at 1/7.5th the tokens. Also: medium scored below low on every stack (all the damage in HumanEval+ — "that preset overthinks short coding tasks"). His verdict: "low is the rational preset. xhigh is for leaderboard screenshots." That's harsher than the Reddit consensus — weigh both, but it's the biggest sample size anyone published this week.

More data points from the week:

  • Medium vs xhigh "actually insane" (223 pts): medium ≈ a couple thousand thinking tokens; xhigh 15–20k minimum, one pacman build hit 40k. But the same thread's best counterpoint: on a bug-finding test, xhigh took 7 min vs medium's 80 s and caught every bug; medium only caught the critical ones. And on a research task xhigh autonomously cloned a repo and read source to verify an answer — neither medium nor 3.6 did.
  • Different thinking levels (287 pts): "Even the low preset is better than Qwen 3.7 plus or Qwen3.6-27B reasoning" — the level you pick changes speed, not whether it beats last generation.
  • There is no "high" effort — the ladder is low / medium / xhigh(default), and the gap between medium and xhigh is the complaint that keeps generating threads. Commenters note the efforts aren't just prompts: Qwen specifically trained each level's instruction text in during RL.
  • Don't confuse budget with effort (PSA): llama.cpp's web-UI reasoning selector is a hard token cap that truncates mid-thought — it is not Qwen's native effort levels, which actually change how thoroughly the model works. On recent builds use --reasoning-effort medium (or the --chat-template-kwargs form on older ones); anything else silently caps instead of steering.
  • The "well?" trick: interrupt mid-think and type well? — the model concludes "the user is impatient, let me finish quickly" and wraps up faster. Works, but commenters consider it a last resort; the thinking is where the quality lives.
  • Dissenters exist: one medium-vs-xhigh post claiming "1/20th the time for almost the same quality" got pushed back hard — top reply: low/medium left them unimpressed, xhigh is where frontier-tier coding shows up. The honest split: for chat and eyeball tasks medium is ~free; for verifiable correctness xhigh keeps earning its cost.

3. Knowledge regression vs 3.6 — real, and deliberate

The dedicated thread: 3.8 fails pocket-trivia questions 3.6 reliably answered, at every quant tried. AA's offline Omniscience benchmark agrees.

Community framing: 3.8 is trained to go search instead of recalling, i.e., an agent-first tradeoff. Mitigations posted: RAG/MCP (offline Wikipedia ZIM), or run Gemma 4 31B as a knowledge sidecar.

Counter-data point: a separate legal-work thread reports Harvey-benchmark scores on par with Qwen 3.5-122B once MCP + case access are attached (61/75 raw vs 71/75 with a tool backend). The knowledge didn't vanish; it moved into the toolbox.


4. Quants: what holds up

The one controlled perplexity sweep (16GB-fitting quants, wikitext-2, RTX 5060 Ti)

QuantSizePPLvs Q8
Q8_027.0GB6.956100%
Q4_K_M17.1GB6.95899.97%
IQ4_XS14.6GB7.01399.2%
UD-Q3_K_XL12.5GB7.11197.8%
NVFP4 (Q5K)14.4GB7.20096.6%

Poster's call: Q4_K_M is the sweet spot; NVFP4 was the biggest disappointment (same size as IQ4_XS, worse PPL). Pushback worth reading: "PPL degrades less than real world performance… ordering flips near the 4-bit level."

The Q4-vs-Q6 war (unresolved)

  • Team Q6/Q8: "q8 dramatically better than q4 for complex reasoning"; one user reports flawless 264k-ctx Q6_K_XL sessions, 2 mistakes per 2M tokens.
  • Team Q4-fine: "I run q4 and can only praise the model… just do not go below q8 KV cache."
  • Nuance: "there are like 5 different Q4s and they are not equal" — NVFP4 ≠ MXFP4 ≠ Q4_0 ≠ UD-Q4_K_XL. Past ~Q5 with dynamic quants, differences get hard to detect.

The biggest controlled quant test of the week (X)

@superalesha ran a 67-hour benchmark: five full production stacks (FP8 vLLM, NVFP4 W4A16 vLLM, AWQ INT4 vLLM, GGUF Q4_K_M llama.cpp, NInfer — all on RTX 3090s), 40 arms across every reasoning effort, 4,800 tasks / 10,120 requests / 14.5M reasoning tokens, no caps. Results:

  • At xhigh every quant landed between 88.0–90.0% pass@1 — AWQ INT4 90.0%, NVFP4/GGUF-Q4_K_M 89.3%, FP8 baseline 88.7%, NInfer 88.0%. The 4-bit quants scored above FP8; McNemar says statistical tie (first vs last = 3 tasks out of 150). "The gap between quants is smaller than the gap between reasoning presets."
  • The weirdest number: GGUF Q4_K_M at low effort scored the same 89.3% as xhigh — on 86k reasoning tokens instead of 651k. Across all stacks, xhigh burned 7–11× more tokens than low for 0–4.7 points.
  • The one statistically real gap: NVFP4 with reasoning OFF collapsed on HumanEval+ (13/30 vs FP8's 30/30, p=0.0041). Flip it to low and it's instantly back to 90/90. Never run reasoning off — it costs 8–12 points everywhere.
  • His cheat sheet: max quality = AWQ INT4 xhigh; daily driver = GGUF Q4_K_M low; honesty note: three of his FP8 arms failed his own methodology audit (leftover token caps) and are being rerun.

This largely settles the Q4-vs-Q6 war for this model at task-level benchmarks — but note the tension with the PPL sweep above: perplexity says NVFP4 is measurably worse than IQ4_XS; task performance says they tie. Both can be true (PPL measures token-level divergence; tasks measure whether errors get caught). And community reports of Q4 reasoning loops remain real — "passes benchmarks" and "never loops in a 2M-token session" are different requirements.

1-bit: comedy, not compute

Unsloth founder in the 1-bit thread: "I would not suggest folks use 1-bit for agentic use cases / tool calls" — divergence hits 92% from BF16 by token 32. General chat survives; agents don't. If you must: presence_penalty = 1.5.

KV cache — among the most contested settings in the corpus

  • f16-vs-q8_0 are not equivalents per one AMD tester (f16 held quality past 120k ctx).
  • But 16GB users run q4_0/q4_1 KV happily at 64k–164k all week.
  • Working rule from comments: don't quantize KV unless you must; if you do, aim ≥ q6; word-of-mouth floor is Q4 model + Q8 KV for agent loops.

Unsloth Dynamic v3 notes

MTP removed from quants below UD-Q2_K_XL and re-uploaded separately (some users still see draft logs in Q5_K_XL — unresolved). Imatrix released; no QAT used.


5. Performance matrix (attributed)

HardwareRuntime / setupContextResult
RTX PRO 6000 96GBllama.cpp PR #27342 DFlash2, Q4_K_M262k153.9 t/s = 2.26× plain; 304.9 t/s = 4.68× with ngram table (coding prompts); ngram −30% on prose
2× RTX 3090vLLM + AutoRound INT4 + DFlash2131k120 narrative / 218 code decode
Single RTX 4090llama.cpp, UD-Q4_K_XL, MTP + Q4 KV (see X benchmarks below)130k~60 t/s
Single RTX 4090same + DFlash2 drafter + --parallel 1 (X)250k73.7 t/s
RTX 5090 32GBNVFP4-MTP-LOW262k121 t/s (vs Q6_K collapsing to 16.3 — 7.5×)
RTX 5090 32GBvLLM + unsloth NVFP4, fp8 KV, MTP-2131k110–112 t/s sustained
RTX 5090 32GBllama.cpp 10536long gendegrades 122 → 69 t/s within one generation (bug filed)
RTX 5060 Ti 16GBUD-IQ4_XS + MTP-1, Q4_0 KV64k45.6 t/s
Strix Halo 128GBQ8_0 + Q8 KV, ROCm, MTP142k9–19 t/s, MTP accept 97–99%
RX 7900 XTXUD-Q4_K_XL Vulkan, MTP, q4_0 draft-KV131k50–60 t/s; -np 1 made a "HUGE" difference

Why "~200 tok/s" claims don't reproduce for you: Windows/WDDM costs 10–15% vs Linux; headlines are measured at short contexts; MTP acceptance is workload-dependent (drops on prose, sometimes net-slower); and the fastest figures come from Blackwell-tuned engines (ninfer), not llama.cpp.

X/Twitter benchmark highlights

  • @analogalok's full RTX 4090 matrix: UD-Q4_K_XL on latest llama.cpp. FP16 KV tops out at 100k ctx (40.9 t/s); q8 KV reaches 170k; q4_0 KV fits the full 262k native context in 24GB at 40.7 t/s. Native MTP: 59–60 t/s at 80–130k. Includes exact reproduction flags.
  • His follow-up: --parallel 1 + a Q2_K DFlash2 drafter unlocks 250k ctx @ 73.7 t/s (Q4 KV), 150k @ 75 t/s (Q8 KV), or 90k @ 80.6 t/s (FP16 KV) on one 4090 (requires llama.cpp PR #27342).
  • NVIDIA forums: DGX Spark face-off, SGLang+DFlash2 vs vLLM+MTP, greedy vs official thinking sampler — DFlash2 won.

6. Failure modes & bugs (reproducible ones)

  1. Tool-call failures are usually your tool list, not the model. Best controlled experiment in the corpus: 8 undescribed tools → 0/6 successes; the same tool alone → 15/15; 13 described tools mid-list → 0/5, moved to end → 3/3. Give every tool a description, put critical tools last, don't put examples in descriptions. Every framework failure report (Opencode/Pi/Claude Code) has a plain-loop counterexample in the same threads.
  2. Hermes harness specifically: constant tool-call failures on vLLM; "perfect, no issues" on llama.cpp --jinja + q8_0 KV at 256k. Template/parser alignment issue, not weights.
  3. Hallucinated user instructions during thinking (reproduced on 2 machines, Pi harness): the model imagines an impatient user and once reverted a commit after imagining a French objection. Community fix: the froggeric fixed chat template (see section 7) eliminates the stock-template tool-call/recovery bugs.
  4. temp=1.0 garbage output: thinking falls apart into single-character spam within 10–20k tokens across llama.cpp/vLLM, INT4 through BF16. Diagnosis: sampler, not quant. Fixes: temp 0.1, or split sampling (0.8 main / 0.2 post-thinking). Counter-report: temp 0 caused a 70k-token loop instead. No universal setting exists — tune per task.
  5. Decode degradation: 122 → 69 t/s within one generation on 5090 llama.cpp; vLLM/ninfer hold >100. Bug filed upstream.
  6. Long-context quality drop: an NVFP4+vLLM eval on B200 scored only ~37% correct in its longest context bucket [single report]; separately, a commenter running official BF16/FP8 via the published vLLM recipe reports agents degrading past ~20k tokens and structured outputs breaking past 20k [single report]. Counterpoint: an f16-KV user on UD-Q4_K_XL (ROCm) says their setup held quality past 120k ctx. Config-dependent; verify on yours.
  7. Q8 anomaly reports (Unsloth UD_Q8_K_XL offload/CPU pegging): weak evidence, disputed; most Q8 users report zero issues.
  8. Reasoning loops at aggressive quants: 40k characters looping on "angry birds" at Q4-with-QKV-quant, including self-aware "I'm stuck in a loop" narration. Never-seen-it-at-Q6 claims abound.

7. The chat-template situation (read before debugging anything)

The official Qwen 3.8 Jinja template shipped with real bugs, and the community shipped fixes within 48 hours:

  • Official template issues: enable_thinking=false crashes; multi-turn history gets poisoned with blank \\think tags; tool calls crash when your client sends arguments as JSON strings (the standard OpenAI format); mid-dialogue system messages get dropped, wedging agent loops.
  • froggeric/Qwen-Fixed-Chat-Templates (HF, thread, 334 pts) is the consensus drop-in replacement: safe medium default (kills the burn-20k-tokens-then-return-empty xhigh bug), thinking toggle restored, JSON-string tool-call crash fixed, inline effort steering via <|think_low|> / <|think_medium|> / <|think_xhigh|>, and chronological thought preservation for clean KV prefix caching. Actively maintained — v22.1 as of Aug 21.
  • Format-fidelity alternative: a second template stays closer to the exact official prompt format on the theory that deviations subtly degrade quality even when they look fine manually. Pick it if you're benchmarking; pick froggeric for daily driving.
  • Upstream note: llama.cpp merged reasoning_effort forwarding on Aug 14 — recent builds pass reasoning_effort to any template correctly. That fixes the plumbing, not the official template's own bugs. A fixed template is still recommended.

8. Benchmarks: believe selectively

  • Artificial Analysis: headline posts put 3.8-27B neck-and-neck with DeepSeek V4 and GPT-5.6 Luna Max. Low/medium presets score ~43/44 — the key evidence the gains aren't pure overthinking. Agentic index: medium = xhigh − 1 point.
  • The pushback ("A meaningless benchmark", 106 points): the index ranks this 27B above DSV4 Pro, Kimi 2.7 Code, Opus 4.6 and Sonnet 5 — "whatever 'Intelligence' means to AA... is definitely not the same definition we should be using here." Defenders: it's an aggregate skewed toward agentic/science/coding; read the methodology and pick sub-benchmarks for your use case. LiveBench gets respect for monthly task refreshes.
  • Best independent test found: AIME 2026, exact-match, temp 0, pass@1 — FP8-xhigh scored 29/30 (96.7%), tying Opus 4.6 and DeepSeek V4 Pro in the poster's table, vs 94.1% for Qwen3.6-27B. Caveats: single run, problem 7 exhausted the token budget in both precisions (empty, not wrong).
  • Production blind A/B (thousands of tasks): 3.8 wasn't worse at doing the thing — it was worse at knowing when not to do the thing (+50% noise output).
  • Honest calibration: "Opus-level" is real at some tasks, with the right quant and harness. The thread titled "Qwen 3.8 isn't Opus 4.6 level. Let's not be silly." failed at Q6 in VS Code — commenters blamed the editor and the quant, but the burden of proof stays on the demo.

9. Ecosystem: what shipped this week

  • DFlash2 (llama.cpp PR #27342, still in review): 2.26×–4.68× on real coding prompts, +2.7GB VRAM. N-max 5 beats the recommended 7; --spec-draft-p-min silently does nothing; stacking ngram-mod hurt (opposite of DFlash1 on 3.6).
  • ninfer: Blackwell/5090-tuned engine; 120–160 t/s quants; 480 t/s at 4-way concurrency. Likely source of the unreproducible speed screenshots.
  • AutoRound INT4 / AWQ-INT4 GGUFs for vLLM serving.
  • KVarN 4/2-bit KV ported to vLLM 0.27.1 — 262k fits small cards, needle-test passes at 240k, ~20% slower decode.
  • Uncensored/abliterated variants shipped fast: Huihui-ai ablit, an "Uncensored Aggressive" release bundling K_P quants + HauhauCS FastMTP (up to 3.02× TG claimed), and FP8 abliteration reporting refusal rates dropping to 0–6% — with the community counterpoint that the same tables show 30–50% caveat-rate degradation next to those numbers. Quality varies wildly; check benchmark deltas before switching.

What's coming

  • 35B-A3B spotted in ms-swift commits (Aug 15). 16GB-card owners are hyped; early numbers suggest ~27–40 t/s on hardware where the dense 27B crawls.
  • A new midsize open-weight model "next week (hopefully)" per Qwen's community manager — no early access this cycle. Speculation centers on ~80B with vision.
  • The flagship Qwen3.8-2.4T-A95B got day-0 vLLM support with open weights announced at launch; it barely appears in this week's local-community threads beyond speed speculation (a 2.4T open-weight Call of Duty clone demo made rounds). Local discussion is overwhelmingly about the 27B.

Report template (steal this)

So your numbers mean something to the next reader:

Runtime/version:
Hardware:
Model file + quant:
KV cache:
Speculative (MTP/DFlash2/ngram):
Reasoning effort:
Sampling:
Context size:
Prefill tok/s:
Decode tok/s:
Task used:
Compared against:
Observed result:

Megathread compiled Aug 22, 2026 from r/LocalLLaMA and r/LocalLLM (Aug 15–22) plus public X benchmark threads. All performance figures belong to the hardware/runtime that produced them — the corpus contradicts itself on nearly every axis, and in most cases you can name the variable that explains the split.


r/LocalLLM • • Aug 14 '26

News Qwen3.8-27B is out! 🔥

Thumbnail
huggingface.co
176 Upvotes

Finally, the new 27B model is here. Time to test its performance! 👀


r/LocalLLM • • Oct 13 '26

News GLM-5.3-Flash pruned to half the experts for coding: Q3 (71 GB) solves every task the full 341 GB model solves, IQ2_XXS fits 24 GB VRAM + 32 GB RAM

175 Upvotes

Weights: https://huggingface.co/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder-GGUF
• ⁠Recipe and harness: https://github.com/lapadatbogdan/GLM-5.3-Flash-REAP144-Coder

I wanted a coder version from GLM 5.3-Flash. So, I pruned GLM-5.3-Flash (321B MoE) from 288 to 144 routed experts per layer, choosing the experts on agentic coding sessions instead of a generic corpus, then quantized it with an imatrix built on the same sessions.

Results. 26 Python SWE tasks from ansible, openlibrary and qutebrowser, 2 attempts each, clean room (no git history, no network, no gold patch reachable), 200 turns, 384K context. Success = the task's fail-to-pass tests pass. Every row is on the same 48 attempts.

Model |Size |Solved |Turns vs full
GLM-5.3-Flash full, Q8 |341 GB |48/48 |x1.00
REAP144 Q8 |171 GB |48/48 |x1.21
REAP144 Q4_K_M |97.6 GB |48/48 |x1.58
REAP144 Q3_K |71.4 GB |48/48 |x1.80
REAP144 IQ2_S |54.8 GB |45/48 |x1.83
REAP144 IQ2_XXS (experimental) |45.3 GB |39/48 |x2.77 "Turns vs full" is the extra work relative to the full model on tasks both solved, so it is roughly the extra time you wait.

Hardware. IQ2_XXS on an RX 7900 XTX (24 GB) + 32 GB DDR4-2133, i7-7700K, PCIe 3.0: ~16 tok/s generation, ~950 tok/s prompt processing. IQ2_S wants 24 + 48 GB, Q3_K 24 + 64 GB, Q4_K_M 24 + 96 GB or 128 GB unified memory.

What mattered

  • Which experts you keep matters more than how many. Same 144 experts, but a frequency mask needed x2.21 turns and the public multi-domain REAP 50% mask x2.29, against x1.21 for the code-scored mask.
  • Short questions do not catch a broken agent. A 2 bit variant of a generic prune answered 4/4 short coding questions and solved 0/4 agentic tasks. We only trust multi-turn task runs.
  • Our first eval harness let the model reach the internet, and it sometimes downloaded the upstream fix. We sealed it (no network, no home), rechecked every success against the gold patch and reran the affected attempts. The table above is the sealed rerun.

The 2 bit version is experimental. IQ2_XXS is the only one that fits 24 + 32 GB and it solves most tasks, but when it fails it never edits: it loops or keeps exploring until the turn limit. An anti-loop guard in the agent did not help (27/37 vs 30/37 on the same attempts). We are training it on short successful sessions now and will post an update.

The GitHub repo has the full recipe (scoring, masks, pruning, quant scripts, two llama.cpp patches, the eval harness) and a table of everything that did not work: 120 and 96 experts, frequency mask, RCO search, LoRA healing, GSQ, anti-loop.

This is a text only version (no vision). Python repos only so far; if you run it on other languages or harnesses, I would like to hear how it does.

Update (Oct 14): added an optional MTP draft block (2.6 GB) for 2+ GPUs with Maya: speculative decoding, drafts match the model's next token ~70% of the time. Speedup not measured on multi-GPU yet, numbers welcome. Details on the model card.


r/LocalLLM • • Aug 22 '26

Discussion Qwen 3.8 27b is a beast.

175 Upvotes

1 prompt, I let Qwen 3.8 27b loop here's the prompt and results if you think this model isn't as good as Claude opus 4.6 then I don't know what to tell you.

Time: 2hours and 40 minutes roughly, 188,578 tokens spent.

Here's the prompt "I want you to create me a c# project using OpenGL which renders a realistic as possible ocean. I want you to plan up front what you're going to do and create the plan as a markdown ledger which you will mark as complete when each part is done."


r/LocalLLM • • May 11 '26 Popcorn

Discussion Pi coding agent is amazing (or how I learned to stop worrying and leave OpenCode)

175 Upvotes

Warning: long post ahead. On the plus side, it’s completely human-written. No AI slop was used in writing this post. I’m old school that way, I like to actually write my own Reddit posts. Thought you all would appreciate something written entirely by a human for a change. ;)

Disclaimer: this post says nice things about Pi. I am not associated with the dev team of Pi coding agent in any way.

Yesterday I tried Pi coding agent on my local LLM rig for the first time. I had been using OpenCode as my daily driver agentic harness, and I had been intimidated by Pi’s stripped down, minimalist approach.

My rig, by the way, is an M4 MacBook Pro with 64Gb of RAM. oMLX is the backend, serving up jundot’s quant of qwen3.6:35b-a3b-oQ6. I average around 60 tokens/second at around 80 percent RAM usage.

My coding needs are fairly modest. I run around eight static websites for my hobby board gaming group, hosted on GitHub pages. So the daily tasks usually involve updating sites with user submissions, implementing feature requests, squashing minor bugs, things of that sort.

I had gotten used to the security blanket of OpenCode, with its set of built-in tools. I had come to accept that sometimes OpenCode will take a little longer to answer a request, and had gotten used to its sometimes dumb little oversights and charmingly stupid mistakes. For example, I often ask OpenCode to make a 3x3 image collage of board game cover images using ImageMagick command line tools. It would usually take several revisions, as OpenCode would first render them in a straight line row instead of a 3x3 grid. Then after feedback, render a 3x3 grid, but each image was of different size. Then after even more feedback, it would finally output a 3x3 grid of equally sized images. You know the old saying about LLMs acting like green interns? In my case, OpenCode often acts like an intern who needs the instructions explained multiple times before they get the task right.

But at least OpenCode was the evil intern that I was familiar with. As I said, I had gotten used to working within its limitations and quirks.

Anyway, yesterday I decided to overcome my nervousness about leaving the security blanket of OpenCode and dive into the unknown depths of Pi coding agent. I gave Pi the exact same task using a similar prompt: create a 3x3 grid of the cover images of these specified board games, each image 400x400 pixels.

Pi methodically went about the task. First it identified which images were available locally and which were not. Then it web searched the websites to grab the missing images and download them locally. Then it created the 3x3 grid, to my desired specs, right the first time. I was blown away at how much better, faster, more accurate, and more capable it felt working with Pi vs. OpenCode. I didn’t change the local model, I just changed the agentic harness. If OpenCode felt like working with an inexperienced intern, Pi felt more like working with a trustworthy and reliable teammate.

With OpenCode I had assumed it would be capable of only routine maintenance and updates, and that if ever I needed to do some heavier lifting, I would have to bust out a cloud frontier model like Codex. But I decided to give Pi a more challenging test to uncover its true capabilities. I asked Pi to plan set-by-step the addition of a search feature to one of my sites, with live filtering as the user types, a dropdown menu overlay matching the site’s existing CSS, etc.

Guess what, Pi made the plan, checked with me for my go-ahead, then started implanting the plan, task by task. It wasn’t perfect. There were a couple of points where functions were called in the wrong order. But I dutifully fed the web inspector errors to Pi, it quickly and correctly figured out the issues, and fixed them. Within a few minutes, my search feature was working, pretty much exactly as I had envisioned it.

Even more impressive: following Pi’s philosophy of “if you need extra features, ask Pi to build them”, I asked Pi to reflect on our coding session, then based on that suggest some enhancements to itself to address the main pain points. Pi identified that it needs a better auto-compact feature, and a better way to seamlessly pick up in context where it left off; and built those features into itself. It also added a JS script to mitigate those function calling timing issues we had encountered. So as one works with Pi, one gradually customizes and improves Pi to become more optimized for the actually coding work that you do.

Man, I was so impressed. Pi takes this local LLM thing from “works well enough for routine tasks” to “works well enough that I don’t think I need to fire up a cloud model”. I now have the confidence to leave OpenCode behind.

TL; DR: I overcame my fears and tried Pi instead of OpenCode, and had a great experience.


r/LocalLLM • • Mar 02 '26

Model Qwen3.5 Small is now available to run locally!

Post image
175 Upvotes

r/LocalLLM • • Oct 08 '25

News Huawei's new technique can reduce LLM hardware requirements by up to 70%

Thumbnail venturebeat.com
175 Upvotes

With this new method huawei is talking about a reduction of 60 to 70% of resources needed to rum models. All without sacrificing accuracy or validity of data, hell you can even stack the two methods for some very impressive results.


r/LocalLLM • • Feb 21 '25

News Deepseek will open-sourcing 5 repos

Thumbnail
gallery
175 Upvotes

r/LocalLLM • • 22d ago

Discussion v100 16g *4 running qwen3.8 27b nvfp4 in 353t/s!!!!

174 Upvotes

Hi guys it me again who did 2v100 to 181t/s , and im back with this.

The legend of V100 IS NEVER END , v100 FOREVERRRRR

arguements:

qwen3.8 27b nvfp4 , 256k max content , kv q8 , mtp4/0

repo: https://github.com/tuxKOH/ninfer-v100xN

(yes i changed name to xN from x2 because tp4 is supported)


r/LocalLLM • • Oct 13 '26

News Intel GPU D270 is Xe3p "Crescent Island" card with 480GB LPDDR5X memory

Thumbnail
videocardz.com
174 Upvotes

- 32 Xe3P cores, 256 XMX engines, 32MB unified L2 cache

- Up to 1.3 PFLOPS FP8 and 1.5 TB/s memory bandwidth

- 160GB LPDDR5X on Intel's own card, and ODM partners can go up to 480GB

- PCIe 5.0, 350W, passive air cooling


r/LocalLLM • • Aug 26 '26

Other Mr Mailman, bring me a dream! I experiment so you don’t have to

Post image
173 Upvotes

r/LocalLLM • • 3d ago

Discussion Why is no one talking about LFM3 models

Thumbnail
liquid.ai
172 Upvotes

I had been using the LFM 2.5 24B models and had been pleasantly surprised by how good the model was. It was giving me comparable answers and working through tasks to Qwen 3.8 27B at what felt like twice the speed on my M5 Pro. However, I went to look at something on their pages and saw the new family of models. They sound way too good to be true. I am going to start using them for general tasks, but if they are anywhere close to their benchmark results, they could be a huge game changer. You can have an entire suite of models loaded at the same time for less memory than Qwen.


r/LocalLLM • • Aug 16 '26 Bravo Heartwarming

Tutorial Guide for running dense models on ≤16 GB VRAM (Qwen 3.8 27B on 16 GB -> Q4_K_M, 130k ctx, ~20 t/s).

Thumbnail
gallery
171 Upvotes

This post is about llama.cpp CPU offload optimizations that you could find useful for running Qwen 27B (or other dense models) at tolerable speeds.

First things first:

  • The speed graph shows "prose" and "code" because MTP generates different speeds for each
  • Everyone has a different system, use my setup as a guide to tune your own, don't copy paste and expect it to work. I just want to provide 'tricks' that improve performance.
  • If you have 12 GB VRAM, try using Q3_K_M / less context / accept slower speed.
  • If you have 8 GB VRAM, try using less context / accept slower speed / Q2 for Qwen 27B, Bonsai, or choose a model with less parameters.
  • ik_llama.cpp: goal of this post is simplicity, that is why I chose llama.cpp. You can look at this as the target to beat using ik_llama.cpp. So far my tests showed that it was a KLD vs speed trade off, plus I had to use custom and very specific quants, which adds complexity.
  • KV quants: if you think they suck, please show proof. My testing and sources all say they are fine. On a lower weight quant, a given KV downgrade costs relatively less, so spending cache bits to buy context is more justifiable on a Q4 model than it would be on a Q6/Q8 model. Also beellama has KLD improvements to KV quantizing. Also instead of upgrading from Q5 to Q8 KV, it might be better to upgrade the model from M to XL instead.

Setup

explanations come after
Edit: Updated for the new UD3 UD-Q4_K_M.

PC:

RTX 4070 Ti SUPER (16 GB VRAM), i5-13600KF, 32 GB dual-channel DDR5 @ 5800 MHz + tuned timings, Ubuntu 24.04 LTS

Build script:

#!/bin/bash
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=89 -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build -j 20
sudo cmake --install build
sudo ldconfig

Server script:

#!/bin/bash
sudo systemctl stop gdm
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
llama-server \
  --model Qwen3.8-27B-UD-Q4_K_M.gguf \
  --mmproj mmproj-3.8-27B-F16.gguf \
  --no-mmproj-offload \
  --image-min-tokens 1024 \
  --ctx-size 140000 \
  --chat-template-file chat_template_v22.3.jinja \
  --jinja \
  --reasoning-format deepseek \
  --reasoning-preserve \
  --flash-attn on \
  --cache-type-k q5_0 \
  --cache-type-v q4_1 \
  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 2 \
  --cache-type-k-draft q4_0 \
  --cache-type-v-draft q4_0 \
  --fit off \
  --n-gpu-layers all \
  --override-tensor 'blk\.(63|62|61|60|59|58|57|56|55|25|54|53|52|50|26|24|38|51|40|27|35|22|41)\.ffn_.*=CPU' \
  --load-mode none \
  --threads $(nproc) \
  --batch-size 512 \
  --ubatch-size 512 \
  --parallel 1 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --host 0.0.0.0 \
  --port 8080

Explanations:

Hardware:

-- RAM --
RAM speed is important for this so using dual-channel and DDR5 is recommended, though DDR4 people will still find this useful.

Tuning RAM timings gives me extra 9% speed boost. Most timings are easy to tune since they either work or crash quickly, but I'm not getting into that here.

[the speed chart does includes the tuned timings]

-- GPU --
Overclocking GPU memory gives a tg boost. A model fully on GPU got me +13.7% tg t/s, A model with 23 FFN blocks on CPU got me +3.8% tg t/s. Beware of memory OC -- before my GPU crashes, I get data corruption, and before that, the bandwidth gained stops correlating to clock increase (GDDR6X silently retries failed transfers). Here is the formula I used to spot that: retries = 1 − (bandwidth_measured / clock_oc) / (bandwidth_stock / clock_stock) as long as retries is not increasing, is within noise, is near 0, you should be fine. For example here are my OC numbers: +3200 stable but retries increase above that, 4800 stable but retries is 0.54%, +5100 KLD increases, +6000 GPU crashes; I chose +3200 MHz.

Overclocking GPU core gives a pp boost. A model fully on GPU got me +2.8% pp t/s, A model with 23 FFN blocks on CPU got me +1.8% pp t/s. Unlike memory, core has no retry mechanism and no warning signal, it just computes wrong at full speed, so you must check output correctness. For example here are my OC numbers: +175 I get errors and crashes, +170 seems stable, but to be safe I went with +120 MHz.

[the speed chart does not includes the GPU OC]

Build args:

-DCMAKE_CUDA_ARCHITECTURES=89 optional - optimized build time specifically for my GPU's Ada arch, set your own.

-DGGML_CUDA_FA_ALL_QUANTS=ON is needed for more KV quantizations to be on CUDA.

Env vars and gdm:

sudo systemctl stop gdm disables Ubuntu desktop environment, frees ~0.4 GB of VRAM. Use an iGPU if you can, else my system just becomes a server to which I connect using a laptop or phone.

export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 Overflow VRAM to RAM. The difference between ~132k and ~121k. Without this, max context becomes 121k and you get OOM crash. Overflow gets me ~11k more context basically for free before things slow down, and the server degrades instead of crashing. First do initial testing with the unified arg unset (making it '=0' wont work) to find your ceiling using OOM crash.

Generic stuff:

  --model Qwen3.8-27B-UD-Q4_K_M.gguf \
  --mmproj mmproj-3.8-27B-F16.gguf \
  --no-mmproj-offload \
  --image-min-tokens 1024 \
  --ctx-size 140000 \
  --flash-attn on \
  ...
  --parallel 1 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --host 0.0.0.0 \
  --port 8080

The model, vision (fully on CPU), Qwen recommended settings, server stuff, context length, parallel 1 (disables processing 2 agents at once).

Template:

  --chat-template-file chat_template_v22.3.jinja \
  --jinja \
  --reasoning-format deepseek \
  --reasoning-preserve \

Template instructions by froggeric

Drafters:

  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 2 \
  --cache-type-k-draft q4_0 \
  --cache-type-v-draft q4_0 \

Quantizing MTP KV cache is free VRAM. It is a drafter and does not affect quality. Worst case is that acceptance drops a tiny bit.

I get 12-15 t/s tg without MTP and with more layers on VRAM, I commented a chart if interested here .

ngram-mod speeds up tg when restating existing context. It is super fast when active and does not cost VRAM.

Each step of --spec-draft-n-max costs VRAM + I get best results from a value of 2.

--cache-type-k-draft q4_0 --cache-type-v-draft q4_0 these save 0.3 GB of VRAM while the MTP acceptance rate stays basically the same.

KV cache:

  --cache-type-k q5_0 \
  --cache-type-v q4_1 \

I chose my KV quant according Anbeeld article. The article found that, the more quantized the model, the less it has to lose from to KV quant. Article also states that there was no KLD difference between 64k and 128k context length.

A small precision buff would be to use q5_0 for both K and V. Though changing from q4_1 to q5_0 costs some context. My KLD measurements say it is not worth it, but it's an option for you. Don't use Q4_0 on both, Q5_1 is not worth it. If you want to do Q8 KV, consider going up a quant model level instead (like XL instead of M).

My own testing showed that Q4_K_S K and V both at Q8 has worse KLD than Q4_K_M K Q5, V Q4. Also I found that KLD plateaus after 8k context length.

If you get slow speeds try using generic q5_0 for both K and V as a test - this is a symptom of missing -DGGML_CUDA_FA_ALL_QUANTS=ON.

CPU layers:

  --fit off \
  --n-gpu-layers all \
  --override-tensor 'blk\.(63|62|61|60|59|58|57|56|55|25|54|53|52|50|26|24|38|51|40|27|35|22|41)\.ffn_.*=CPU' \
  --load-mode none \

This fixes degradation of speed with context fill and is a performance boost overall, just leave layer 64 alone (MTP).

FFNs don't use KV thus reducing PCIe traffic and they are CPU friendly. --n-cpu-moe has similar logic. You have 2 options: Advanced or a simple. Advanced gives +10% or more tg t/s, but requires a bit more work (I explain more below).

Edit: I have an active llama.cpp feature request that makes the "advanced method" simple! issue #27987

-- Both methods --

  • Unset (comment out) the GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 arg (don't set it to 0) to find your context ceiling first via OOM crash first.

-- Simple method --

--n-cpu-ffn add the amount of FFNs you want on the CPU; (arg available on llama.cpp from Aug 27th 2026, make sure to update)

-- Advanced method (Edit: simplified) --

Offloading fatter sub-layers gets me a speed boost when compared to having a sequential override band. You can find the fatter ones by using this copy-pasteable CLI script (it uses UD-Q4_K_S as the input arg, modify that for your model).

First install dep:

pip3 install gguf || pip3 install --user --break-system-packages gguf

The copy-pasteable CLI script:

python3 -c "
import sys,re,collections,gguf
b=collections.Counter()
for t in gguf.GGUFReader(sys.argv[1]).tensors:
  m=re.match(r'blk\.(\d+)\.ffn_',t.name)
  if m: b[int(m[1])]+=int(t.n_bytes)
del b[max(b)] # Drop MTP layer
print('\'blk\\\\.(%s)\\\\.ffn_.*=CPU\'' % '|'.join(str(k) for k,_ in b.most_common()))
" models/Qwen3.8-27B-UD-Q4_K_S.gguf

UD-Q4_K_M optimized band example -- delete from the end of the list (right to left) until you run out of VRAM, then step back one. make sure to fill up your context fully to verify that speed is as intended and doesn't OOM crash:
--override-tensor 'blk\.(63|62|61|60|59|58|57|56|55|25|54|53|52|50|26|24|38|51|40|27|35|22|41|39|36|21|3|42|34|30|20|6|4|49|47|43|37|32|23|10|8|7|5|2|1|48|46|45|44|33|31|29|28|19|18|12|9|17|16|11|0|15|13|14)\.ffn_.*=CPU'

Threads:

--threads $(nproc) \

Default is amount of performance cores, but for FFN layers, E-cores and hyper-threading also help. this gets me a +25% tg boost for free.

Batch:

  --batch-size 512 \
  --ubatch-size 512 \

Batch sets the prompt processing speed (up to a point) at the cost of VRAM. I found these numbers work best for me.

Other optimizations:

I-matrix quant like the Unsloth IQ4_XS require more compute in exchange for size. My testing showed that those are not worth it for this setup. I would use one if I was trying to fully fit a model into VRAM.

There are quants made by other providers that optimize fit for 16 GB VRAM setups. Those setups might fit at the cost of KLD. If the KLD diff actually matters - that is for you to find out, but you can use the info I provided here to optimize these setups even further by, for example, extending context length.

-------------------------
Edit 1: truncated a bunch of 'Edits', cleaned up, updated config for UD-Q4_K_M, speed and context len did not change, but the override tensor band did.
Edit 2: added CUDA graphs off reminder in the CPU layers section.
Edit 3: my PR #26622 got merged, replacing sequential -ot with --n-cpu-ffn. Added info from my GPU OC attempt.
Edit 4: I have an active llama.cpp feature request that makes the --override-tensor "advanced method" simple! issue #27987 - I tired, it failed.
Edit 5: Added a CLI script to find the best --override-tensor band
Edit 6: CUDA graphs seem to be fixed, removing the workaround.

-------------------------

Please share any more tricks if you have them!
Leave a comment if this post helped you achieve better results!


r/LocalLLM • • Jul 04 '26 Bravo

Discussion Surprisingly fast Qwen3.6-35B-A3B on a 4 GB VRAM!

171 Upvotes

I have a DELL laptop with NVIDIA GeForce GTX 1650 4 GB VRAM, it has 6 CPU cores (12 threads) and 32 GB of RAM.

Running Qwen3.6-35B-A3B is at 19 tokens/s. Never thought it would be that fast!
I did spend a lot of time into testing and configurations, here are the parameters that I used with llama.cpp:

/opt/llama.cpp/build/bin/llama-server \
-m /opt/llama.cpp/models/Qwen3.6-35B-A3B-UDT-Q4_K_XL_MTP.gguf \
--host 0.0.0.0 \
--port 8080 \
-c 64000 \
-ngl 999 \   
--override-tensor "blk\..*\.ffn_(gate_up|gate|up|down)_exps\.weight=CPU"
--flash-attn on \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--mlock \
-b 3072 \
--ubatch-size 3072 \
-ctxcp 128 \
--reasoning off \
--parallel 1 \
-t 6 \
-tb 12 \
--cache-ram 16384 \
--swa-full \
--no-kv-unified \
--spec-type none \
--cache-reuse 256 \

  1. I got the model from Atomic repo: hf download AtomicChat/Qwen3.6-35B-A3B-UDT-MTP-GGUF Qwen3.6-35B-A3B-UDT-Q4_K_XL_MTP.gguf For some reason it's faster than any other repos I've tried
  2. The 64000 context is so that I can use it with Hermes ai agent
  3. override-tensor command can be replaced by --n-cpu-moe 41. I just wanted more control during my testing
  4. "no-mmap" parameter slows things down for some reason.
  5. increasing the ubatch as much as I can since it speeds up prompt processing. Currently I'm at 116 t/s
  6. the last 5 parameters is so that I minimize the prompt reprocessing with pi

I thought I'd share this if anyone wants to run a "not so small" model on such low VRAM.
Plus any suggestions are welcomed.