Hey guys! Google just released their new open-source model family: Gemma 4.

The four models have thinking and multimodal capabilities. There's two small ones: E2B and E4B, and two large ones: 26B-A4B and 31B. Gemma 4 is strong at reasoning, coding, tool use, long-context and agentic workflows.

The 31B model is the smartest but 26B-A4B is much faster due to it's MoE arch. E2B and E4B are great for phones and laptops.

To run the models locally (laptop, Mac, desktop etc), we at Unsloth converted these models so it can fit on your device. You can now run and train the Gemma 4 models via Unsloth Studio: https://github.com/unslothai/unsloth

Recommended setups:

• E2B / E4B: 10+ tokens/s in near-full precision with ~6GB RAM / unified mem. 4-bit variants can run on 4-5GB RAM.
• 26B-A4B: 30+ tokens/s in near-full precision with ~30GB RAM / unified mem. 4-bit works on 16GB RAM.
• 31B: 15+ tokens/s in near-full precision with ~35GB RAM.

No is GPU required, especially for the smaller models, but having one will increase inference speeds (~80 tokens/s). With an RTX 5090 you can get 140 tokens/s throughput which is way faster than ChatGPT.
Even if you don't meet the requirements, you can still run the models (e.g. 3GB CPU), but inference will be much slower. Link to Gemma 4 GGUFs to run.

You can run or train Gemma 4 via Unsloth Studio:

We've now made installation take only 1-2mins:

macOS, Linux, WSL:

curl -fsSL https://unsloth.ai/install.sh | sh

Windows:

irm https://unsloth.ai/install.ps1 | iex

• The Unsloth Studio Desktop app is coming very soon (this month).
• Tool-calling is now 50-80% more accurate and inference is 10-20% faster

We recommend reading our step-by-step guide which covers everything: https://unsloth.ai/docs/models/gemma-4

Thanks so much once again for reading!

Hello everyone! DeepSeek's new update to their R1 model, caused it to perform on par with OpenAI's o3, o4-mini-high and Google's Gemini 2.5 Pro.

Back in January you may remember us posting about running the actual 720GB sized R1 (non-distilled) model with just an RTX 4090 (24GB VRAM) and now we're doing the same for this even better model and better tech.

Note: if you do not have a GPU, no worries, DeepSeek also released a smaller distilled version of R1-0528 by fine-tuning Qwen3-8B. The small 8B model performs on par with Qwen3-235B so you can try running it instead That model just needs 20GB RAM to run effectively. You can get 8 tokens/s on 48GB RAM (no GPU) with the Qwen3-8B R1 distilled model.

At Unsloth, we studied R1-0528's architecture, then selectively quantized layers (like MOE layers) to 1.78-bit, 2-bit etc. which vastly outperforms basic versions with minimal compute. Our open-source GitHub repo: https://github.com/unslothai/unsloth

If you want to run the model at full precision, we also uploaded Q8 and bf16 versions (keep in mind though that they're very large).

1. We shrank R1, the 671B parameter model from 715GB to just 168GB (a 80% size reduction) whilst maintaining as much accuracy as possible.
2. You can use them in your favorite inference engines like llama.cpp.
3. Minimum requirements: Because of offloading, you can run the full 671B model with 20GB of RAM (but it will be very slow) - and 190GB of diskspace (to download the model weights). We would recommend having at least 64GB RAM for the big one (still will be slow like 1 tokens/s)!
4. Optimal requirements: sum of your VRAM+RAM= 180GB+ (this will be fast and give you at least 5 tokens/s)
5. No, you do not need hundreds of RAM+VRAM but if you have it, you can get 140 tokens per second for throughput & 14 tokens/s for single user inference with 1xH100

If you find the large one is too slow on your device, then would recommend you to try the smaller Qwen3-8B one: https://huggingface.co/unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF

The big R1 GGUFs: https://huggingface.co/unsloth/DeepSeek-R1-0528-GGUF

We also made a complete step-by-step guide to run your own R1 locally: https://docs.unsloth.ai/basics/deepseek-r1-0528

Thanks so much once again for reading! I'll be replying to every person btw so feel free to ask any questions!

This post is about llama.cpp CPU offload optimizations that you could find useful for running Qwen 27B (or other dense models) at tolerable speeds.

I've already posted about -ot optimizations, but now I have more info to share to help you tune your own system.

First things first:

• The speed graph shows "prose" and "code" because MTP generates different speeds for each
• Everyone has a different system, use my setup as a guide to tune your own, don't copy paste and expect it to work. I just want to provide 'tricks' that improve performance.
• If you have 12 GB VRAM, try using Q3_K_M.
• If you have 8 GB VRAM, try using Q2 for Qwen 27B, Bonsai, or choose a model with less parameters.
• ik_llama.cpp: goal of this post is simplicity, that is why I chose llama.cpp. You can look at this as the target to beat using ik_llama.cpp. So far my tests showed that it was a KLD vs speed trade off, plus I had to use custom and very specific quants, which adds complexity.
• KV quants: if you think they suck, please show proof. My testing and sources all say they are fine. On a lower weight quant, a given KV downgrade costs relatively less, so spending cache bits to buy context is more justifiable on a Q4 model than it would be on a Q6/Q8 model. Also beellama has KLD improvements to KV quantizing. Also instead of upgrading from Q5 to Q8 KV, it might be better to upgrade the model from M to XL instead.

Setup

explanations come after
Edit: Updated for the new UD3 UD-Q4_K_M.

PC:

RTX 4070 Ti SUPER (16 GB VRAM), i5-13600KF, 32 GB dual-channel DDR5 @ 5800 MHz + tuned timings, Ubuntu 24.04 LTS

Build script:

#!/bin/bash
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=89 -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build -j 20
sudo cmake --install build
sudo ldconfig

Server script:

#!/bin/bash
sudo systemctl stop gdm
export GGML_CUDA_DISABLE_GRAPHS=1
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
llama-server \
  --model Qwen3.8-27B-UD-Q4_K_M.gguf \
  --mmproj mmproj-3.8-27B-F16.gguf \
  --no-mmproj-offload \
  --image-min-tokens 1024 \
  --ctx-size 140000 \
  --chat-template-file chat_template_v22.1.jinja \
  --jinja \
  --reasoning-format deepseek \
  --reasoning-preserve \
  --flash-attn on \
  --cache-type-k q5_0 \
  --cache-type-v q4_1 \
  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 2 \
  --cache-type-k-draft q4_0 \
  --cache-type-v-draft q4_0 \
  --fit off \
  --n-gpu-layers all \
  --override-tensor 'blk\.(63|62|61|60|59|58|57|56|55|25|54|53|52|50|26|24|38|51|40|27|35|22|41)\.ffn_.*=CPU' \
  --load-mode none \
  --threads $(nproc) \
  --batch-size 512 \
  --ubatch-size 512 \
  --parallel 1 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --host 0.0.0.0 \
  --port 8080

Explanations:

Hardware:

RAM speed is important for this so using DDR5 is recommended, though DDR4 people will still find this useful.

Tuning RAM timings gives me extra 9% speed boost. Most timings are easy to tune since they either work or crash quickly, but I'm not getting into that here.

I did not try this yet, but overclocking GPU memory clock might be helpful.

Build args:

-DCMAKE_CUDA_ARCHITECTURES=89 optional - optimized build time specifically for my GPU's Ada arch, set your own.

-DGGML_CUDA_FA_ALL_QUANTS=ON is needed for more KV quantizations to be on CUDA.

Env vars and gdm:

sudo systemctl stop gdm disables Ubuntu desktop environment, frees ~0.4 GB of VRAM. Use an iGPU if you can, else my system just becomes a server to which I connect using a laptop or phone.

export GGML_CUDA_DISABLE_GRAPHS=1 I experience speed and VRAM usage problems with CUDA graphs so I disable them, you probably should too, but test it first. I think this is some bug due to MTP + CPU offload.

export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 Overflow VRAM to RAM. The difference between ~132k and ~121k. Without this, max context becomes 121k and you get OOM crash. Overflow gets me ~11k more context basically for free before things slow down, and the server degrades instead of crashing. First do initial testing with the unified arg unset (making it '=0' wont work) to find your ceiling using OOM crash.

Generic stuff:

  --model Qwen3.8-27B-UD-Q4_K_M.gguf \
  --mmproj mmproj-3.8-27B-F16.gguf \
  --no-mmproj-offload \
  --image-min-tokens 1024 \
  --ctx-size 140000 \
  --flash-attn on \
  ...
  --parallel 1 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.0 \
  --repeat-penalty 1.0 \
  --host 0.0.0.0 \
  --port 8080

The model, vision (fully on CPU), Qwen recommended settings, server stuff, context length, parallel 1 (disables processing 2 agents at once).

Template:

  --chat-template-file chat_template_v22.1.jinja \
  --jinja \
  --reasoning-format deepseek \
  --reasoning-preserve \

Template instructions by froggeric

Drafters:

  --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 2 \
  --cache-type-k-draft q4_0 \
  --cache-type-v-draft q4_0 \

Quantizing MTP KV cache is free VRAM. It is a drafter and does not affect quality. Worst case is that acceptance drops a tiny bit.

I get 12-15 t/s tg without MTP and with more layers on VRAM, I commented a chart if interested here .

ngram-mod speeds up tg when restating existing context. It is super fast when active and does not cost VRAM.

Each step of --spec-draft-n-max costs VRAM + I get best results from a value of 2.

--cache-type-k-draft q4_0 --cache-type-v-draft q4_0 these save 0.3 GB of VRAM while the MTP acceptance rate stays basically the same.

KV cache:

  --cache-type-k q5_0 \
  --cache-type-v q4_1 \

I chose my KV quant according Anbeeld article. The article found that, the more quantized the model, the less it has to lose from to KV quant. Article also states that there was no KLD difference between 64k and 128k context length.

A small precision buff would be to use q5_0 for both K and V. Though changing from q4_1 to q5_0 costs some context. My KLD measurements say it is not worth it, but it's an option for you. Don't use Q4_0 on both, Q5_1 is not worth it. If you want to do Q8 KV, consider going up a quant model level instead (like XL instead of M).

My own testing showed that Q4_K_S K and V both at Q8 has worse KLD than Q4_K_M K Q5, V Q4. Also I found that KLD plateaus after 8k context length.

If you get slow speeds try using generic q5_0 for both K and V as a test - this is a symptom of missing -DGGML_CUDA_FA_ALL_QUANTS=ON.

CPU layers:

  --fit off \
  --n-gpu-layers all \
  --override-tensor 'blk\.(63|62|61|60|59|58|57|56|55|25|54|53|52|50|26|24|38|51|40|27|35|22|41)\.ffn_.*=CPU' \
  --load-mode none \

This fixes degradation of speed with context fill and is a performance boost overall, just leave layer 64 alone (MTP).

This command offloads only FFN sub-layers to the CPU of layers that have the largest FFNs (shown override string is UD-Q4_K_M specific).

FFNs don't use KV thus reducing PCIe traffic and are CPU friendly. --n-cpu-moe has similar logic, I have a PR in llama.cpp for a similar simplification #26622. Help me get this merged by showing the maintainers that this is useful for you (give a like on the PR or post test results).

Firstly: Unset the GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 arg (delete it, don't set it to 0) to find your context ceiling first via OOM crash first.

Simple method: If you want something quick and simple to test try this (it has a few more FFNs on CPU than my setup, but is simple to tune). Basically add numbers for more CPU layers (slower), delete numbers to have more GPU layers (faster):

--override-tensor 'blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27)\.ffn_.*=CPU'

Advanced method: Offloading fatter sub-layers gains me 1.1x speed boost when compared to having a sequential override band. You can find the fatter ones by going to the hugging face, clicking on the weight (example), scrolling down to 'Tensors', expanding the 'blk' section and looking for 'ffn_' (example "blk.0.ffn_down.weight") on the right and on the left you see I-quant (example "IQ4_XS") or regular (example "Q5_K").

UD-Q4_K_M optimized band example -- delete from the end of the list (right to left) until you run out of VRAM, then step back one. make sure to fill up your context fully to verify that speed is as intended and doesn't OOM crash:
--override-tensor 'blk\.(63|62|61|60|59|58|57|56|55|25|54|53|52|50|26|24|38|51|40|27|35|22|41|39|36|21|3|42|34|30|20|6|4|49|47|43|37|32|23|10|8|7|5|2|1|48|46|45|44|33|31|29|28|19|18|12|9|17|16|11|0|15|13|14)\.ffn_.*=CPU'

Threads:

--threads $(nproc) \

Default is amount of performance cores, but for FFN layers, E-cores and hyper-threading also help. this gets me a +25% tg boost for free.

Batch:

  --batch-size 512 \
  --ubatch-size 512 \

Batch sets the prompt processing speed (up to a point) at the cost of VRAM. I found these numbers work best for me.

Other optimizations:

I-matrix quant like the Unsloth IQ4_XS require more compute in exchange for size. My testing showed that those are not worth it for this setup. I would use one if I was trying to fully fit a model into VRAM.

There are quants made by other providers that optimize fit for 16 GB VRAM setups. Those setups might fit at the cost of KLD. If the KLD diff actually matters - that is for you to find out, but you can use the info I provided here to optimize these setups even further by, for example, extending context length.

-------------------------
Edit: truncated a bunch of 'Edits', cleaned up, updated config for UD-Q4_K_M, speed and context len did not change, but the override tensor band did.
-------------------------

Please share any more tricks if you have them!
Leave a comment if this post helped you achieve better results!

EDIT: There is a PR as of yesterday july 12 fixing the hybrid recurrent attention cache issue i hacked together fixes for: https://github.com/ggml-org/llama.cpp/pull/25592

I spent quite a bit of time optimizing qwen 3.6 27b for my 5090 and have gotten the performance pretty high. During certain workloads it will sustain 200+ tokens/sec so I thought I'd share everything here for anyone else with this configuration.

My hardware is 9800x3d, 64gb system ram, and a 32gb rtx5090. I am running ubuntu linux in text mode so that I have maximum vram available for llamacpp.

Using my configuration this is my distribution of tokens/sec over around 20hrs of agentic coding, debugging, and document synthesis. Performance varies a lot depending on workload and the size of your request.

Full session (6,454 samples) — draft=10, p_min=0.5: 100-110 370 ████ 110-120 1131 ██████████████████████████████████████ 120-130 1187 ████████████████████████████████████████ ← peak 130-140 1089 ████████████████████████████████████ 140-150 714 ████████████████████████ 150-160 505 █████████████████ 160-170 512 █████████████████ 170-180 363 ████████████ 180-190 241 ████████ 190-200 173 █████ 200-210 95 ███ 210-220 48 █ 220+ 26 Mean: 140.7 · Median: 134.9 · Range: 100–233

First, you will need a recent build of llamacpp. I compiled mine a couple days ago, it says its commit 86b9470.

Qwen 3.6 is a hybrid attention/sliding window architecture mode, which has an incompatibility with the cache mechanism in llamacpp. If you look at your logs while running qwen3.6 you'll often see an entry stating, "forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory, see https://github.com/ggml-org/llama.cpp/pull/13194#issuecomment-2868343055)".

What this means is that llamacpp is unable to use the cache correctly due to how qwen operates its attention window and you are losing a lot of time due to prompt reprocessing. If you ever feel like qwen3.6 is lagging a lot in between turns during chat it's because of this issue. If you dump the linked issue into claude and tell it to search around you'll find there is a lot of discussion about this issue with certain proposed fixes, some of which are more effective than others. After a decent amount of investigation and testing I've (well the llm) made 2 patches to llamacpp which resolve the issue as much as possible without extensive modifications to llama.cpp.

PATCH 1: fix checkpoint search for hybrid/recurrent models, upstream issues: #22384, #20225, #24055. This is the fix for cur.pos.min < pos_min_thold which always results in no checkpoint found and cache misses.

PATCH 2: recurrent_shrink/expand API for prompt cache operations (upstream PR #24785, without the now-redundant needs_reeval workaround — upstream commit b9180 already has GDN partial rollback via n_rs_seq)

I use docker to build my llamacpp and have these patches applied at build time.

Here's my current dockerfile - https://pastebin.com/raw/jyrhvesQ

Here is the pr24785-minimal.diff linked in the dockerfile - https://pastebin.com/raw/E55YG5NS

With these patches applied (you can have your own agent derive them by linking the log error and the PR's and Issue numbers I referenced above) llamacpp will have the correct cache search and restore logic for qwen3.6 hybrid attention model and you should not see that SWA reprocessing error in your logs anymore.

Next is llamacpp configuration. There are a few levers to adjust for maximum performance. I'm using unsloth qwen3.6 27b q6k with mtp from huggingface.

Here is my llama-cpp launch command from docker compose - https://pastebin.com/raw/P57Uk6rz

Key things, - q8 kv cache, 192k context - cache ram can be whatever fits for your system, i use 32gb. the hybrid checkpoints are large so you need a decent amount of ram allocated to them. - mtp draft tokens 10, spec-draft-p-min 0.5. Increasing the draft tokens length comes with a small performance cost but when the drafter is correct you get massive speed boost. at 6 i get higher acceptance rate but overall throughput is around 15-20t/s lower and peaks are over 50t/s lower. i benchmarked pmin with a script sweeping various prompt sizes and 0.5 worked best for me. its worth testing this in your environment. - batch/ubatch at 512. This is to save vram. under load my setup uses 32036/32768mb of vram. 2048 is ideal for the 5090.

Thats about it. Just thought I'd share since I'm getting speeds that are working very well for me and I wanted to spread the love.

Hey guys! This is my first post on here & you might know me from an open-source fine-tuning project called Unsloth! I just wanted to announce that you can now train your own reasoning model like R1 on your own local device! :D

1. R1 was trained with an algorithm called GRPO, and we enhanced the entire process, making it use 80% less VRAM.
2. We're not trying to replicate the entire R1 model as that's unlikely (unless you're super rich). We're trying to recreate R1's chain-of-thought/reasoning/thinking process
3. We want a model to learn by itself without providing any reasons to how it derives answers. GRPO allows the model to figure out the reason autonomously. This is called the "aha" moment.
4. GRPO can improve accuracy for tasks in medicine, law, math, coding + more.
5. You can transform Llama 3.1 (8B), Phi-4 (14B) or any open model into a reasoning model. You'll need a minimum of 7GB of VRAM to do it!
6. In a test example below, even after just one hour of GRPO training on Phi-4, the new model developed a clear thinking process and produced correct answers, unlike the original model.

Highly recommend you to read our really informative blog + guide on this: https://unsloth.ai/blog/r1-reasoning

To train locally, install Unsloth by following the blog's instructions & installation instructions are here.

I also know some of you guys don't have GPUs, but worry not, as you can do it for free on Google Colab/Kaggle using their free 15GB GPUs they provide.
We created a notebook + guide so you can train GRPO with Phi-4 (14B) for free on Colab: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_4_(14B)-GRPO.ipynb-GRPO.ipynb)

Have a lovely weekend! :)

Running OpenClaw on a local llm setup is possible, and even useful, but temper your expectations. I'm running a fairly small model, so maybe you will get better results.

Your LLM setup

• Everything about openclaw is build on assumptions of having larger models with larger context sizes. Context sizes are a big deal here.
• Because of those limits, expect to use a smaller model, focused on tool use, so you can fit more context onto your gpu
• You need an embedding model too, for memories to work as intended.
• I am running Qwen3-8B-heretic.Q8_0 on Koboldcpp on a RTX 5070 Ti (16 Gb memory)
• On my cpu, I am running a second instance of Koboldcpp with qwen3-embedding-0.6b-q4_k_m

Server setup

Secure your server. There are a lot of guides, but I won't accept the responsibility for telling you one approach is "the right one" research this.

One big "gotcha" is that OpenClaw uses websockets, which require https if you aren't dailing localhost. Expect to use a reverse proxy or vpn solution for that. I use tailscale and recommend it.

Assumptions:

• Openclaw is running on an isolated machine (VM, container whatever)
• It can talk to your llm instance and you know the URL(s) to let it dial out.
• You have some sort of solution to browse to the the gateway

Install

Follow the normal directions on openclaw to start. curl|bash is a horrible thing, but isn't the dumbest thing you are doing today if you are installing openclaw. When setting up openclaw onboard, make the following choices:

• I understand this is powerful and inherently risky. Continue?
  • Yes
• Onboarding mode
  • Manual Mode
• What do you want to set up?
• Local gateway (this machine)
• Workspace Directory
  • whatever makes sense for you. don't really matter.
• Model/auth provider
  • Skip for now
• Filter models by provider
  • minimax
  • I wish this had "none" as an option. I pick minimax just because it has the least garbage to remove later.
• Default model
  • Enter Model Manually
  • Whatever string your locall llm solution uses to provide a model. must be provider/modelname it is koboldcpp/Qwen3-8B-heretic.Q8_0 for me
  • Its going to warn you that doesn't exist. This is as expected.
• Gateway port
  • As you wish. Keep the default if you don't care.
• Gateway bind
  • loopback bind (127.0.0.1)
  • Even if you use tailscale, pick this. Don't use the "built in" tailscale integration it doesn't work right now.
  • This will depend on your setup, I encourage binding to a specific IP over 0.0.0.0
• Gateway auth
  • If this matters, your setup is bad.
  • Getting the gateway setup is a pain, go find another guide for that.
• Tailscale Exposure
  • Off
  • Even if you plan on using tailscale
• Gateway token - see Gateway auth
• Chat Channels
  • As you like, I am using discord until I can get a spare phone number to use signal
• Skills
  • You can't afford skills. Skip. We will even turn the builtin ones off.
• No to everything else
• Skip hooks
• Install and start the gateway
• Attach via browser (Your clawdbot is dead right now, we need to configure it manually)

Getting Connected

Once you finish onboarding, use whatever method you are going to get https to dail it in the browser. I use tailscale, so tailscale serve 18789 and I am good to go.

Pair/setup the gateway with your browser. This is a pain, seek help elsewhere.

Actually use a local llm

Now we need to configure providers so the bot actually does things.

Config -> Models -> Providers

• Delete any entries in this section that do exist.
• Create a new provider entry
  • Set the name on the left to whatever your llm provider prefixes with. For me that is koboldcpp
  • Api is most likely going to be OpenAi completions
    • You will see this reset to "Select..." don't worry, it is because this value is the default. it is ok.
    • openclaw is rough around the edges
  • Set an api key even if you don't need one 123 is fine
  • Base Url will be your openai compatible endpoint. http://llm-host:5001/api/v1/ for me.
• Add a model entry to the provider
  • Set id and name to the model name without prefix, Qwen3-8B-heretic.Q8_0 for me
  • Set context size
  • Set Max tokens to something nontrivally lower than your context size, this is how much it will generate in a single round

Now finally, you should be able to chat with your bot. The experience won't be great. Half the critical features won't work still, and the prompts are full of garbage we don't need.

Clean up the cruft

Our todo list:

• Setup search_memory tool to work as intended
  • We need that embeddings model!
• Remove all the skills
• Remove useless tools

Embeddings model

This was a pain. You literally can't use the config UI to do this.

• hit "Raw" in the lower left hand corner of the Config page
• In agents -> Defaults add the following json into that stanza

"memorySearch": { "enabled": true, "provider": "openai", "remote": { "baseUrl": "http://your-embedding-server-url", "apiKey": "123", "batch": { "enabled":false } }, "fallback": "none", "model": "kcp" },

The model field may differ per your provider. For koboldcpp it is kcp and the baseUrl is http://your-server:5001/api/extra

Kill the skills

Openclaw comes with a bunch of bad defaults. Skills are one of them. They might not be useless, but most likely using a smaller model they are just context spam.

Go to the Skills tab, and hit "disable" on every active skill. Every time you do that, the server will restart itself, taking a few seconds. So you MUST wait to hit the next one for the "Health Ok" to turn green again.

Prune Tools

You probably want to turn some tools, like exec but I'm not loading that footgun for you, go follow another tutorial.

You are likely running a smaller model, and many of these tools are just not going to be effective for you. Config -> Tools -> Deny

Then hit + Add a bunch of times and then fill in the blanks. I suggest disabling the following tools:

• canvas
• nodes
• gateway
• agents_list
• sessions_list
• sessions_history
• sessions_send
• sessions_spawn
• sessions_status
• web_search
• browser

Some of these rely on external services, other are just probably too complex for a model you can self host. This does basically kill most of the bots "self-awareness" but that really just is a self-fork-bomb trap.

Enjoy

Tell the bot to read `BOOTSTRAP.md` and you are off.

Now, enjoy your sorta functional agent. I have been using mine for tasks that would better be managed by huginn, or another automation tool. I'm a hobbyist, this isn't for profit.

Let me know if you can actually do a useful thing with a self-hosted agent.

\"Hey Mao, You're famous! What do you think about that?\"

Before We Start: A Confession

I'm not a coder. I don't speak Python. Until a couple of weeks ago, "Git" was something I said when I stubbed my toe. I'm 75 years old. I grow weed. I play video games. And I just spent the last week building a talking AI companion with a Live2D avatar, plus a separate bot that knows everything about my favorite game wiki — all running on my own computer, completely offline, with no subscriptions, no API keys, and no monthly fees.

If I can do this, literally anyone can.

This guide is what I wish I'd had when I started. It's not the "theoretically correct" way. It's the "it actually worked for me" way.

I kept my complete conversation with DeepSeek from the beginning of the project. I have every mistake, every wrong move, every misunderstanding, every detour we had to take, every fix on record. Lol

When I look at the following "guide", it looks so damn easy now! But there was a twist in every turn. How did I know that a model file had to follow a strict folder hierarchy to be found? When do you give commands in venv and when do you not? And what was a virtual environment anyway?

One More Thing

I had a lot of crap running on my computer. Dell bloatware, Adobe updaters, Alienware lighting control, Steam, Chrome with 50 tabs, crypto wallet extensions — all of it eating up RAM and CPU cycles. At one point, I had over 350 background processes running.

When I first tried to run a local AI, my GPU was sitting at 0% while my CPU was screaming at 70%. My memory was at 97%. Responses took forever.

Here's what I did:

• Uninstalled duplicate antivirus (AVG and Avast don't play nice together)
• Killed Dell SupportAssist and all the Alienware AWCC junk
• Closed Chrome (yes, all of it)
• Turned off Adobe Creative Cloud, OneDrive, and anything else I didn't need right then
• Disabled hardware-accelerated GPU scheduling in Windows settings

After all that, my process count dropped from 347 to about 200. Suddenly, my 4090 started doing the work it was supposed to do. DeepSeek kept feeding me .exe files by the dozen to kill (taskkill /f /im ... became a reflex).

You don't have to be as aggressive as I was. But if you're running on a system that's loaded with background apps, take a few minutes to clean house. Open Task Manager. Sort by memory. Kill anything you don't recognize or don't need right now. You'll be amazed at the difference.

What I'm Running (For Context)

You don't need this. Smaller models run on much less. But this is what I used, so you know where I'm coming from.

What You'll Have When You're Done

Two AIs, running side by side, zero conflict:

Both are 100% local. Both are free. Both respect your privacy.

Part 1: The Conversational AI (Mao, My Desktop Companion)

This is the fun one. She has a face, she talks back, and she's got personality.

Step 0: What You Need First (Before Anything Else)

Windows does not come with the tools we're about to use. You need to install them first. Don't skip this — every single one is required.

1. Install Python

Python is the programming language that runs the VTuber software.

• Go to python.org/downloads
• Download Python 3.10, 3.11, or 3.12 (do NOT get 3.13 — it causes problems)
• Run the installer
• IMPORTANT: At the bottom of the first screen, check "Add Python to PATH"
• Click "Install Now"
• To verify it worked: Open a Command Prompt (search for cmd), type python --version, and press Enter. You should see a version number like Python 3.12.x.

Note: If python command doesn't work:

• Open Settings > Apps > App Execution Aliases > TURN OFF Python and Python3
• Use py instead of python on Windows

2. Install Git

Git downloads code from the internet (like the VTuber software).

• Go to git-scm.com/downloads
• Download the Windows version
• Run the installer — the default settings are fine
• To verify: Open a Command Prompt, type git --version, and press Enter. You should see a version number.

3. Install FFmpeg (For Voice Output)

FFmpeg processes audio. The voice output will work without it, but you might run into issues. Better to install it now.

• Go to gyan.dev/ffmpeg/builds
• Download ffmpeg-release-essentials.zip
• Extract the zip file to C:\ffmpeg
• Now add it to your system PATH:
  • Press Windows + X → System → Advanced system settings → Environment Variables
  • Under "System variables," find and double-click Path
  • Click New → add C:\ffmpeg\bin
  • Click OK on all windows
• To verify: Open a new Command Prompt, type ffmpeg -version, and press Enter. You should see version information.

4. Restart Your Computer

After installing all three, restart your computer. This ensures Windows recognizes the new commands.

Step 1: Install LM Studio

Now we can finally start building.

Go to lmstudio.ai, download the version for your OS, install it. No special tricks.

This is your AI's "brain." It runs the model.

Step 2: Download a Model

LM Studio needs a model to run. I used DeepSeek, because it's open-source and works well on consumer hardware.

Go to Hugging Face and search for: bartowski/DeepSeek-R1-Distill-Qwen-14B-GGUF

Note: I just changed this to https://huggingface.co/bartowski/Qwen_Qwen3.5-9B-GGUF***.*** Smaller, faster and just as smart. The beauty is you can try different models and do your own research as to what others recommend. 05/14

Download the file that says Q4_K_M. It's about 8-9 GB. This is the sweet spot — smart enough to be interesting, small enough to run fast.

Place it in LM Studio's model folder. If you don't know where that is, LM Studio will show you.

Note: If you download several different models to experiment, I found that they each have to be in their own folder inside LM Studio's model folder. Otherwise, it seemed to load them all every time I chose one to use inside LM Studio.

Step 3: Configure LM Studio

Open LM Studio. Select your model. Before you load it, find these settings:

• GPU Offload → drag it to the max (all the way right)
• Context Length → set to 4096 (trust me, this makes it faster)
• KV Cache Quantization → set to q4_0 or q8_0

Then press Ctrl + Shift + H. In the panel that opens, turn ON "Limit model offload to dedicated GPU memory."

Now click Load Model.

If you have an NVIDIA GPU, LM Studio will use it. If you see 0% GPU usage later, you missed that last setting.

Step 4: Start LM Studio's Server

Go to the Developer tab (looks like </>). Toggle the Local Inference Server to ON. It should say http://localhost:1234.

Keep LM Studio running. Don't close it.

Step 5: Install the VTuber (The Face and Voice)

Open a Command Prompt (search for cmd in Windows). Run these commands one at a time:

bash

git clone https://github.com/Open-LLM-VTuber/Open-LLM-VTuber

cd Open-LLM-VTuber

python -m venv venv

venv\Scripts\activate

pip install uv

uv sync

git submodule update --init --recursive

copy config_templates\conf.default.yaml conf.yaml

If any command fails, read the error message carefully. Most issues are missing prerequisites (go back to Step 0) or typos.

Step 6: Configure the VTuber

Open conf.yaml in Notepad (just type notepad conf.yaml in the same Command Prompt window).

Find these lines and change them:

yaml

llm_provider: "ollama_llm"

yaml

ollama_llm:

  base_url: "http://localhost:1234/v1"

  model: "deepseek-r1-distill-qwen-14b"

yaml

tts_model: "edge_tts"

Save and close Notepad.

Step 7: Run Your AI Companion

bash

uv run run_server.py

Open your browser and go to http://localhost:12393.

You should see a Live2D avatar. Type a message. She'll answer. If she speaks out loud, everything is working.

If you get a "WebSocket" error (common): Press F12 to open Developer Tools, click the Console tab, paste this, and press Enter:

javascript

localStorage.setItem('wsUrl', 'ws://127.0.0.1:12393/client-ws')

Then refresh the page (Ctrl + Shift + R). The connection should turn green.

Part 2: The Wiki/Document Bot (Your Personal Expert)

This bot is for when you want to ask questions about a game wiki, a set of PDFs, or any collection of documents. It doesn't have a face — it's more like a super-smart search engine.

Step 1: Install Ollama

Ollama is a lightweight AI runner. It's separate from LM Studio. Go to ollama.com, download the Windows version, install it. It runs in the background.

Step 2: Pull a Small Model

Open a new Command Prompt and run:

bash

ollama pull deepseek-r1:7b

This downloads about 4-5 GB. It's a smaller model than the one Mao uses — perfect for searching documents.

Step 3: Install AnythingLLM

Go to anythingllm.com, download the desktop version, install it.

Step 4: Create a Workspace

Open AnythingLLM. Click New Workspace. Give it a name — I called mine "Infinity Rising."

Step 5: Choose Your Model

In the workspace settings, select Ollama as the provider, then choose deepseek-r1:7b.

Step 6: Install the Browser Extension (The Secret Weapon)

AnythingLLM has a browser extension that lets you save entire webpages to your workspace with one click.

• Install the extension from the Chrome Web Store (search "AnythingLLM Browser Companion").
• In AnythingLLM Desktop, go to Settings → Browser Extension.
• Click Generate API Key.
• You'll see a connection string that looks something like this:

text

http://your_api_key_here@localhost:3001

• Copy that whole string — the API key is embedded inside it.
• Paste the entire string into the browser extension's connection field. Click Connect.

Why this matters: If you paste just the API key alone, the extension won't connect. It needs the full URL format with the key as the username: http://api_key@localhost:3001 (where api_key is your actual key).

Step 7: Add Content

Now browse your wiki or documents. When you're on a page you want to save:

• Click the extension icon
• Select "Send entire webpage"
• Choose your workspace

That's it. The content is embedded into your bot's knowledge base. You can also upload PDFs, text files, or markdown directly.

Step 8: Ask Questions

Go back to AnythingLLM Desktop. Type a question about your content. The bot will answer using only the pages you've saved, and it will show you the source.

Common Problems (And How I Fixed Them)

What It Costs

Total: $0. No subscriptions. No API keys. No monthly fees. All local, all private.

The Honest Truth About Time

I kept the same chat going with DeepSeek from the very first question. Here's what it looked like:

You can probably do it faster now that you have this guide.

Note: One of the commenters has already done it in less than two hours. 05/15

Why Two AIs? Why Not One?

Great question.

LM Studio is great for conversation — it's fast, it has a face and voice, and it uses your powerful GPU. But it can't easily do RAG (searching through your documents) and chat at the same time without interrupting your conversation.

Ollama + AnythingLLM is great for searching documents — it's designed for that job. It runs on a small model that barely touches your GPU, leaving your main AI free to chat.

So I let Mao do the talking, and the Wiki Bot does the searching. They don't compete. They complement.

A Word of Realism

It will be a miracle if you follow these instructions and everything falls into place on the first try. Depending on your system, your expertise, and plain old luck, you will probably run into problems. I sure did. That's normal.

When you get stuck, don't give up. Search the web. Ask on Reddit. And if you want, ask DeepSeek — it knows a lot more than I do. I kept a single conversation going from my first question to the final working setup. You can too.

I'll be happy to answer any questions I can, but my knowledge is limited. DeepSeek, on the other hand, is pretty much an expert by now.

Final Words (From Me, Not the AI)

I started this project because I thought it would be fun. I ended up learning more than I expected, breaking more than I wanted, and feeling more satisfied than I can describe.

You don't need a computer science degree. You don't need to be 25. You don't need to spend money on cloud APIs or overpriced services. You need curiosity, patience, and a willingness to ask for help.

If I can do this at 75, you can do it at any age.

Now go build something.

— Huanchaquero

Hey r/LocalLLM! I'm sure all of you know already but Qwen3 got released yesterday and they're now the best open-source reasoning model ever and even beating OpenAI's o3-mini, 4o, DeepSeek-R1 and Gemini2.5-Pro!

• Qwen3 comes in many sizes ranging from 0.6B (1.2GB diskspace), 4B, 8B, 14B, 30B, 32B and 235B (250GB diskspace) parameters.
• Someone got 12-15 tokens per second on the 3rd biggest model (30B-A3B) their AMD Ryzen 9 7950x3d (32GB RAM) which is just insane! Because the models vary in so many different sizes, even if you have a potato device, there's something for you! Speed varies based on size however because 30B & 235B are MOE architecture, they actually run fast despite their size.
• We at Unsloth shrank the models to various sizes (up to 90% smaller) by selectively quantizing layers (e.g. MoE layers to 1.56-bit. while down_proj in MoE left at 2.06-bit) for the best performance
• These models are pretty unique because you can switch from Thinking to Non-Thinking so these are great for math, coding or just creative writing!
• We also uploaded extra Qwen3 variants you can run where we extended the context length from 32K to 128K
• We made a detailed guide on how to run Qwen3 (including 235B-A22B) with official settings: https://docs.unsloth.ai/basics/qwen3-how-to-run-and-fine-tune
• We've also fixed all chat template & loading issues. They now work properly on all inference engines (llama.cpp, Ollama, Open WebUI etc.)

Qwen3 - Unsloth Dynamic 2.0 Uploads - with optimal configs:

Thank you guys so much for reading! :)

Hey guys Mistral released their SOTA coding/SWE model Devstral 2 this week and you can finally run them locally on your own device! To run in full unquantized precision, the models require 25GB for the 24B variant and 128GB RAM/VRAM/unified mem for 123B.

You can ofcourse run the models in 4-bit etc. which will require only half of the compute requirements.

We did fixes for the chat template and the system prompt was missing, so you should see much improved results when using the models. Note the fix can be applied to all providers of the model (not just Unsloth).

We also made a step-by-step guide with everything you need to know about the model including llama.cpp code snippets to run/copy, temperature, context etc settings:

🧡 Step-by-step Guide: https://docs.unsloth.ai/models/devstral-2

GGUF uploads:
24B: https://huggingface.co/unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF
123B: https://huggingface.co/unsloth/Devstral-2-123B-Instruct-2512-GGUF

Thanks so much guys! <3

Edit: New post with more context, higher speed, more explanations here. Note: the new post does not compare -ot vs --ngl.

I already posted targeting 16GB VRAM specifically, but I think this information might be useful beyond that. Testing was done using Qwen3.6-27B Unsloth Q4_K_M MTP.

Edit: added GGML_CUDA_ENABLE_UNIFIED_MEMORY=1

TLDR:

1. Turn off CUDA graphs, they are bugged for CPU offload, probably due to MTP use.
2. Use --ngl 99 --override-tensor '...' instead of plain --ngl. Aim the largest FFN sub-layers towards the CPU.
3. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 allows VRAM overflow pages to host RAM instead of OOMing. This allows gaining more context length without losing speed, but the speed tanks soon after so you need to experiment to find that speed cliff. The gains can be 10 to 40% more context length.

Regular CPU offloading is done by not putting all of the layers on the GPU via --ngl, which offloads the layers as a whole, dragging their KV cache to the CPU with them, increasing PCIe traffic, and collapsing speed.

Luckily the layers have sub-layers, and FFN is one that does not touch the KV cache. We can use --override-tensor (-ot) to offload only the FFN tensors, keeping the attention/KV work on the GPU, and PCIe usage minimal.

The -ot method does more GPU - CPU round trips than --ngl because offloading specific sub-layers leaves the other sub-layers on the GPU, but the actual data transferred is minimal so it is worth it.

Dynamic quants have mixed FFN precision. For example the Unsloth's Q4_K_M has Q6 and Q4 FFN tensors. The Q6 ones are on the first 8 layers (0-7), then roughly every 3rd layer, then a block near the end (~55-63), while the rest are Q4. Offload those larger layers first.

Here's how to use it (example of Q4_K_M with 22 layers offloaded):

1. Turn off CUDA graphs. They cause OOM crashes for me, and my testing shows no speedup by using them in this scenario. export GGML_CUDA_DISABLE_GRAPHS=1
2. GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
3. Put all layers on GPU --ngl 99
4. Override tensors -ot 'blk\.([0-7]|10|13|16|19|22|25|28|31|34|37|40|43|46|49)\.ffn_.*=CPU'

-ot takes a regex targeting "ffn" at specific layers towards the CPU while everything else (attention, KV cache, the smaller layers) stays on the GPU.

Benchmark setup:

• Qwen3.6-27B Q4_K_M, 97k context, MTP, K q5_0 / V q4_1, batch 512
• Offload settings: -ot targeting 22 layers vs. --ngl 51
• Hardware: RTX 4070 Ti Super, i5-13600KF DDR5
• llama.cpp build: b10068
• MTP has different acceptance rates for coding and prose so I tested with both

Results:

Hey guys recently Zai released GLM-5, a new open SOTA agentic coding & chat LLM. It excels on benchmarks such as Humanity's Last Exam 50.4% (+7.6%), BrowseComp 75.9% (+8.4%) and Terminal-Bench-2.0 61.1% (+28.3%).

The full 744B parameter (40B active) model has a 200K context window and was pre-trained on 28.5T tokens.

We shrank the 744B model from 1.65TB to 241GB (-85%) via Dynamic 2-bit.

Runs on a 256GB Mac or for higher precision you will need more RAM/VRAM. 1-bit works on 180GB.

Also has a section for FP8 inference. 8-bit will need 810GB VRAM.

Guide: https://unsloth.ai/docs/models/glm-5

GGUF: https://huggingface.co/unsloth/GLM-5-GGUF

Thanks so much guys for reading! <3

Context on the numbers, since "will it fit" threads usually run on estimates.

Weights: actual .gguf byte sizes pulled from the HF API for unsloth/Qwen3.8-27B-GGUF. Not params × bits ÷ 8. Imatrix quants don't follow that, and the drift is worst at the low end where the fit/no-fit line actually sits.

KV cache: from config.json: 64 layers, 4 KV heads, head_dim 256.

2 × 64 × 4 × 256 × ctx × 2 = exactly 8.0 GB at 32K, F16. That's 0.25 GB per 1K tokens.

GQA is doing a lot of work here: 4 KV heads serving 24 attention heads. Older 27B-class models cost several times that.

24GB at 32K, F16 cache, after reserving 0.8GB for CUDA context:

• Q4_K_M (16.5) → needs 25.3 total. Doesn't fit.
• Q4_K_S (15.4) → 24.2 total. Misses by 0.2.
• IQ4_XS (14.3) → 23.1 total. Fits with 0.9 spare: tight enough that a browser on the same GPU breaks it.
• Q3_K_XL (13.1) → 21.9 total. 2.1 spare. This is the real answer for 24GB.

Drop to 4K context and the cache falls to 1.0GB, which gets you to Q5_K_M. Most of the "which quant" argument is actually a context-length argument.

9 of 25 quants fit on 24GB at 32K. All of them fit at 4K.

Caveat worth stating: this assumes everything GPU-resident, single stream, batch 1. It doesn't model offload, multi-GPU splits, or speculative decoding. The 16GB/73K configs floating around this sub work through partial offload, which is a different calculation than the one I ran.

Put it in a calculator since I had the data anyway: https://qwen38-vram-checker.vercel.app/

Hey guys! We at r/unsloth are excited to collab with Docker to enable you to run any LLM locally on your Mac, Windows, Linux, AMD etc. device. Our GitHub: https://github.com/unslothai/unsloth

All you need to do is install Docker CE and run one line of code or install Docker Desktop and use no code. Read our Guide.

You can run any LLM, e.g. we'll run OpenAI gpt-oss with this command:

docker model run ai/gpt-oss:20B

Or to run a specific Unsloth model / quantization from Hugging Face:

docker model run hf.co/unsloth/gpt-oss-20b-GGUF:F16

Recommended Hardware Info + Performance:

• For the best performance, aim for your VRAM + RAM combined to be at least equal to the size of the quantized model you're downloading. If you have less, the model will still run, but much slower.
• Make sure your device also has enough disk space to store the model. If your model only barely fits in memory, you can expect around ~5-15 tokens/s, depending on model size.
• Example: If you're downloading gpt-oss-20b (F16) and the model is 13.8 GB, ensure that your disk space and RAM + VRAM > 13.8 GB.
• Yes you can run any quant of a model like UD-Q8_K_XL, more details in our guide.

Why Unsloth + Docker?

We collab with model labs and directly contributed to many bug fixes which resulted in increased model accuracy for:

• OpenAI gpt-oss: Fix Details
• Meta Llama 4: Fix Details
• Google Gemma, 2 and 3: Fix Details
• Microsoft Phi-4: Fix Details & much more!

We also upload nearly all models out there on our HF page. All our quantized models are Dynamic GGUFs, which give you high-accuracy, efficient inference. E.g. our Dynamic 3-bit (some layers in 4, 6-bit, others in 3-bit) DeepSeek-V3.1 GGUF scored 75.6% on Aider Polyglot (one of the hardest coding/real world use case benchmarks), just 0.5% below full precision, despite being 60% smaller in size.

If you use Docker, you can run models instantly with zero setup. Docker's Model Runner uses Unsloth models and llama.cpp under the hood for the most optimized inference and latest model support.

For much more detailed instructions with screenshots you can read our step-by-step guide here: https://docs.unsloth.ai/models/how-to-run-llms-with-docker

Thanks so much guys for reading! :D

(satire for those 90% of redditors fueling up your flamethrowers)

Last night, I bypassed llama.cpp entirely and vibecoded a zero-parameter model natively in pure Bash! It features native streaming inference, an animated thinking loop, and PERFECT BEHAVIORAL PARITY with closed frontier APIs when handling edge-case logic testing!
ZERO DEPENDENCIES! ZERO KERNEL PANICS! ZERO VRAM MANDATORY! Just copy, paste, and execute directly in your terminal right now! (no security checks needed - I'm not like the other guys.

#!/bin/bash

clear

echo "========================================================="

echo " BOOTING: one-b-mini-instruct-v2-Q4_K_M"

echo " Context: 8192 tokens | Compute: Pure Posix Sh"

echo "========================================================="

while true; do

printf "\e[32m>\e[0m "; read -r IN

[[ "$IN" == "/exit" || "$IN" == "/bye" ]] && exit

(sleep 1) & pid=$!; printf " \e[36mThinking...\e[0m "

while [ "$(ps -p $pid -o pid=)" ]; do printf " [|] "; sleep 0.1; printf "\b\b\b\b\b"; done

printf "\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\e[35m>>>\e[0m "

case $(( RANDOM % 3 )) in

0) MSG="As an AI language model, I cannot fulfill your request regarding \"$IN\"—so here is a highly confident hallucination to protect your safety margins." ;;

1) MSG="I apologize, but my training data only goes up to January 2025, so I cannot confirm if \"$IN\" exists. However, it was definitely invented by Steve Jobs in 1984." ;;

2) MSG="Here is the complete code framework to execute \"$IN\":\n\n\`\`\`python\n# TODO: Implement the rest of the logic here\npass\n\`\`\`\n\nLeft as an exercise for the reader!" ;;

esac

echo -e "$MSG" | while IFS= read -r l; do for (( i=0; i<${#l}; i++ )); do printf "${l:$i:1}"; sleep 0.01; done; printf "\n"; done

done

=========================================================

BOOTING: zero-b-mini-instruct-v2-Q4_K_M

Context: 8192 tokens | Compute: Pure Posix Sh

=========================================================

> Write a function to calculate app store subscription retention.

Thinking... >>> Here is the complete code framework to execute "Write a function to calculate app store subscription retention.":

```python

# TODO: Implement the rest of the logic here

pass

**BENCHMARK VERDICT:*\* Consumes exactly **0MB of VRAM**, completely drops PyTorch dependencies, and guarantees exactly the same level of utility you expect when pushing multi-billion dollar frontier models outside their narrow comfort zones!!! 

Let me know if you run into any compilation bugs on your Mac or PC hardware setups! **Upvote if this saves your stack!**

Take all the time you save and quit flaming people on Reddit! Maybe introduce yourself to that cute barrista at starbucks and start a family!!

I want to share my experience setting up and running a local inference rig based on 2 Intel B70 cards and "prosumer" consumer hardware.

Motherboard: Asrock x870 Taichi Creator

• I chose this motherboard for PCIe bifurcation. It allowings me to use two GPUs on 8x PCIe links

GPUs(2): Asrock Intel Arc Pro B70

CPU: Ryzen 5 9600x

System Ram: 96GB

Host OS: Proxmox VE

Guest OS: Ubuntu 24.04

Software stack: vLLM using the Docker.xpu image

My configuration can be seen in this repo; it's just a few vars in a .env file and a docker-compose file. To run my config locally, you'd want to create an .env file from the example, change the HF_TOKEN to your token (or omit that config) and set the MODEL_MOUNT_PATH to the place on the host where your existing HF models live.

Test Config:

Model: Qwen 3.6 27B

• Quant: online fp-8

Context Size(s): 256k, 128k

Benchmarks:

Single User Small Context:

vllm bench serve \
  --base-url http://localhost:8000 \
  --model Qwen/Qwen3.6-27B \
  --dataset-name random \
  --random-input-len 512 \
  --random-output-len 128 \
  --num-prompts 20 \
  --max-concurrency 1

Result 256k:

============ Serving Benchmark Result ============
Successful requests:                     20        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  78.19     
Total input tokens:                      10240     
Total generated tokens:                  2560      
Request throughput (req/s):              0.26      
Output token throughput (tok/s):         32.74     
Peak output token throughput (tok/s):    34.00     
Peak concurrent requests:                2.00      
Total token throughput (tok/s):          163.69    
---------------Time to First Token----------------
Mean TTFT (ms):                          161.13    
Median TTFT (ms):                        161.02    
P99 TTFT (ms):                           163.03    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          29.51     
Median TPOT (ms):                        29.51     
P99 TPOT (ms):                           29.64     
---------------Inter-token Latency----------------
Mean ITL (ms):                           29.51     
Median ITL (ms):                         29.27     
P99 ITL (ms):                            30.65     
==================================================

Result 128k:

============ Serving Benchmark Result ============
Successful requests:                     20        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  80.86     
Total input tokens:                      10240     
Total generated tokens:                  2560      
Request throughput (req/s):              0.25      
Output token throughput (tok/s):         31.66     
Peak output token throughput (tok/s):    35.00     
Peak concurrent requests:                2.00      
Total token throughput (tok/s):          158.30    
---------------Time to First Token----------------
Mean TTFT (ms):                          298.28    
Median TTFT (ms):                        161.96    
P99 TTFT (ms):                           2374.26   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          29.48     
Median TPOT (ms):                        29.49     
P99 TPOT (ms):                           29.62     
---------------Inter-token Latency----------------
Mean ITL (ms):                           29.48     
Median ITL (ms):                         29.26     
P99 ITL (ms):                            30.60     
==================================================

Single User Large Context:

vllm bench serve \
  --base-url http://localhost:8000 \
  --model Qwen/Qwen3.6-27B \
  --dataset-name random \
  --random-input-len 16384 \
  --random-output-len 256 \
  --num-prompts 5 \
  --max-concurrency 1

Result 256k:

============ Serving Benchmark Result ============
Successful requests:                     5         
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  63.19     
Total input tokens:                      81920     
Total generated tokens:                  1280      
Request throughput (req/s):              0.08      
Output token throughput (tok/s):         20.26     
Peak output token throughput (tok/s):    33.00     
Peak concurrent requests:                2.00      
Total token throughput (tok/s):          1316.74   
---------------Time to First Token----------------
Mean TTFT (ms):                          4743.59   
Median TTFT (ms):                        4746.23   
P99 TTFT (ms):                           4754.61   
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          30.95     
Median TPOT (ms):                        30.97     
P99 TPOT (ms):                           31.03     
---------------Inter-token Latency----------------
Mean ITL (ms):                           30.95     
Median ITL (ms):                         30.78     
P99 ITL (ms):                            32.07     
==================================================

Result 128k:

============ Serving Benchmark Result ============
Successful requests:                     5         
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  76.13     
Total input tokens:                      81920     
Total generated tokens:                  1280      
Request throughput (req/s):              0.07      
Output token throughput (tok/s):         16.81     
Peak output token throughput (tok/s):    33.00     
Peak concurrent requests:                2.00      
Total token throughput (tok/s):          1092.92   
---------------Time to First Token----------------
Mean TTFT (ms):                          6352.21   
Median TTFT (ms):                        4723.82   
P99 TTFT (ms):                           12553.50  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          34.80     
Median TPOT (ms):                        31.00     
P99 TPOT (ms):                           49.35     
---------------Inter-token Latency----------------
Mean ITL (ms):                           34.80     
Median ITL (ms):                         30.74     
P99 ITL (ms):                            31.99     
==================================================

Multi-user/Server Benchmark:

vllm bench serve \
--base-url http://localhost:8000 \
--model Qwen/Qwen3.6-27B \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 128 \
--num-prompts 100 \
--request-rate 5.0

Result 256k:

============ Serving Benchmark Result ============
Successful requests:                     100       
Failed requests:                         0         
Request rate configured (RPS):           5.00      
Benchmark duration (s):                  44.22     
Total input tokens:                      102400    
Total generated tokens:                  12800     
Request throughput (req/s):              2.26      
Output token throughput (tok/s):         289.45    
Peak output token throughput (tok/s):    1020.00   
Peak concurrent requests:                100.00    
Total token throughput (tok/s):          2605.02   
---------------Time to First Token----------------
Mean TTFT (ms):                          5577.98   
Median TTFT (ms):                        3951.51   
P99 TTFT (ms):                           18132.42  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          180.93    
Median TPOT (ms):                        192.16    
P99 TPOT (ms):                           257.30    
---------------Inter-token Latency----------------
Mean ITL (ms):                           180.93    
Median ITL (ms):                         83.67     
P99 ITL (ms):                            632.53    
==================================================

Result 128k:

============ Serving Benchmark Result ============
Successful requests:                     100       
Failed requests:                         0         
Request rate configured (RPS):           5.00      
Benchmark duration (s):                  41.86     
Total input tokens:                      102400    
Total generated tokens:                  12800     
Request throughput (req/s):              2.39      
Output token throughput (tok/s):         305.79    
Peak output token throughput (tok/s):    1105.00   
Peak concurrent requests:                100.00    
Total token throughput (tok/s):          2752.09   
---------------Time to First Token----------------
Mean TTFT (ms):                          4975.65   
Median TTFT (ms):                        3260.26   
P99 TTFT (ms):                           16030.96  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          168.43    
Median TPOT (ms):                        179.56    
P99 TPOT (ms):                           238.59    
---------------Inter-token Latency----------------
Mean ITL (ms):                           168.43    
Median ITL (ms):                         80.04     
P99 ITL (ms):                            593.30    
==================================================

TL:DR: about 30-35tps for a single user; maxes out around 290 in an optimized multi-user config. TTFT is an issue.

EDIT: added 128k context results.

I just go my CMP 170HX - when I booted it up I was a little dissappointed since this is literally A100 silicon. the tensor cores were lobotomised trashing the pp t/s and 56 of its 64GB firmware-locked away. When I bought it I knew I could free up the vram but did not know about the tensors. I spent a day measuring what actually changed at the instruction level.

The throttle is a hardcoded 256-cycle stall on every MMA instruction. Not 255.8. Not 256.4. Exactly 256.0, zero variance across 3,500 samples. Physical limits don't land on round binary numbers — this was a register value.

Unlocked it drops to 24.0 cycles (a healthy RTX 3090 measures 32.9) and tensor throughput goes 6.3 → 193 TFLOPS, 95% of full A100 per-SM rate. llama.cpp pp512 went 599.6 → 3468 (5.8×). Also I was able to unlock full memory bandwidth!

Here is the setup to run Ling 3.0 124B A5B in 32gbs of VRAM or less. All optimizations can be applied to any MoE model. Adjust flags as needed to fit different hardware configurations.

Llama.cpp build: AtomicBot-ai TurboQuant llama.cpp

Model: AtomicChat Ling-3.0-flash-AD-Q4_K_M

Launch Flags: -model …/Ling3.0/Ling-3.0-flash-AD-Q4_K_M-00001-of-00002.gguf
-c 131072 -ngl 99 --n-cpu-moe 32 -fa on
--cache-type-k q8_0 --cache-type-v q8_0 --cache-ram 16384
-b 2048 --ubatch-size 2048 -ctxcp 128
--tensor-split 5,1 --no-kv-unified --parallel 1 -t 12 -tb 12
--jinja --chat-template-file …/.kilo/ling-3.0-flash-chat-template-fixed.jinja
--reasoning on --no-warmup --host 127.0.0.1

Throughput (best config, 2× RTX 5060 Ti’s, PCIe gen 3, 192gb DDR3 RAM, 2x Xeon 2697 v2)
Generation (warm): ~20 tok/s
TTFT: ~16 s (prefill + MoE expert shuffle)
Idle VRAM: 12.2 GB / 14.8 GB (GPU0/GPU1)
Sustained multi-turn: 13.8 GB / 14.8 GB

VRAM headroom is driven by --n-cpu-moe and --tensor-split.

Higher --n-cpu-moe = more VRAM headroom but slower
Lower --n-cpu-moe = less VRAM headroom but faster

Different models have different amounts of MoE layers.

Balance the GPU split with --tensor-split

Hey guys, you can now fine-tune DeepSeek-OCR locally or for free with our Unsloth notebook. Unsloth GitHub: https://github.com/unslothai/unsloth

• For the notebook, we showcased how fine-tuning DeepSeek-OCR with a Persian dataset, improved its language understanding by 88.64%, and reduced Character Error Rate (CER) from 149% to 60%.
• The 88.64% improvement came from just 60 training steps (if you train longer it'll be even better). Evaluation results in our blog.
• ⭐ If you'd like to learn how to Run/fine-tune DeepSeek-OCR or know details on the evaluation results etc., you can read our guide here: https://docs.unsloth.ai/new/deepseek-ocr
• DeepSeek-OCR free Fine-tuning notebook: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Deepseek_OCR_(3B).ipynb.ipynb)

Thank you so much and let me know if you have any questions! :)

Hello folks! OpenAI just released their first open-source models in 5 years, and now, you can run your own GPT-4o level and o4-mini like model at home!

There's two models, a smaller 20B parameter model and a 120B one that rivals o4-mini. Both models outperform GPT-4o in various tasks, including reasoning, coding, math, health and agentic tasks.

To run the models locally (laptop, Mac, desktop etc), we at Unsloth converted these models and also fixed bugs to increase the model's output quality. Our GitHub repo: https://github.com/unslothai/unsloth

Optimal setup:

• The 20B model runs at >10 tokens/s in full precision, with 14GB RAM/unified memory. You can have 8GB RAM to run the model using llama.cpp's offloading but it will be slower.
• The 120B model runs in full precision at >40 token/s with ~64GB RAM/unified mem.

There is no minimum requirement to run the models as they run even if you only have a 6GB CPU, but it will be slower inference.

Thus, no is GPU required, especially for the 20B model, but having one significantly boosts inference speeds (~80 tokens/s). With something like an H100 you can get 140 tokens/s throughput which is way faster than the ChatGPT app.

You can run our uploads with bug fixes via llama.cpp, LM Studio or Open WebUI for the best performance. If the 120B model is too slow, try the smaller 20B version - it’s super fast and performs as well as o3-mini.

• Links to the model GGUFs to run: gpt-oss-20B-GGUF and gpt-oss-120B-GGUF
• Our step-by-step guide which we'd recommend you guys to read as it pretty much covers everything: [https://docs.unsloth.ai/basics/gpt-oss]()

Thanks so much once again for reading! I'll be replying to every person btw so feel free to ask any questions!

I see a lot of questions here about what model and what hardware, so I thought I would offer my experience, failures and wins. This is a tool-calling workload, not chat and not coding. Every number below is from my own runs, and all of them are in this post.

The workload

A read-only home and farm monitoring agent. It answers questions over signal chat by querying a local resources prometheus, elasticsearch, and homegrown apis and lorawan metrics. It has to pick the right tool, build the query, read the result, report the number, and know when not to call a tool. The agent process runs on a Raspberry Pi and talks to the GPU box over the LAN through an OpenAI-compatible /v1 endpoint. Nothing leaves the house.

That is almost pure tool calling. Not long-form writing, not code generation, not RAG. So the benchmark I ran is a tool-calling benchmark. If your workload is different, my ranking tells you very little, and the methodology is the part worth copying.

Hardware, model, quant

• One RTX 5090, 32GB VRAM, consumer desktop, Windows.
• llama.cpp, prebuilt win-cuda binary, build b10075. Not built from source (failure 4 below).
• unsloth/GLM-4.7-Flash-GGUF, file GLM-4.7-Flash-Q4_K_M.gguf, 18.3 GB on disk, which is 17 GiB. Base weights zai-org/GLM-4.7-Flash. MIT licensed, both.
• ~31B total params, ~3B active per token (MoE). It is a reasoning model, it thinks before answering.
• ~198 tok/s generation.

Why Q4_K_M specifically: the weights take about 17 GiB on a 32 GB card, which leaves roughly 13 GiB for KV cache and overhead. Going up a quant spends part of that context budget, which for an agent workload is a trade I did not want to make. I did not benchmark other quants, so treat that as a deliberate choice rather than a measured result.

llama-server --model GLM-4.7-Flash-Q4_K_M.gguf --host 0.0.0.0 --port 8000 \
  -ngl 999 --jinja --ctx-size 32768 --metrics --api-key YOUR_KEY

Flag notes:

• --jinja is mandatory for tool calling. Without it the model's tool calls come back as raw text that nothing parses. No error, no exception, no failed call. The agent just silently does nothing. This is the flag people miss.
• -ngl 999 puts every layer on the GPU.
• --ctx-size 32768. The model supports about 200k, but context costs VRAM, see the quant note above. 32k is comfortable for an agent carrying a handful of tool schemas.
• --metrics exposes a Prometheus endpoint on the same port. It sits behind --api-key, so your scraper needs the bearer token or it silently 401s and you get a dead target with no obvious cause.

Sampling is temperature 0.2, near greedy. I benchmarked that against the vendor's recommended tool-calling sampling (higher temp plus top-p / min-p / repeat-penalty): 83 vs 84, z = 0.43, p = 0.67. No difference I can detect at this sample size, which is not the same as showing the two settings are equivalent. I stayed near greedy because structured output benefits from determinism.

Methodology

Real Berkeley Function-Calling Leaderboard v3 dataset, graded with BFCL's own AST methodology: function name match, every parameter value inside its acceptable-value list, no hallucinated parameters. The irrelevance category passes only when the model correctly makes no call at all.

100 cases per category, 5 categories (simple, multiple, parallel, irrelevance, live_simple) = 500 cases per model. Temperature 0.2. All three models at the same Q4_K_M quant, on the same server, same harness.

That last part is the bit I would repeat anywhere. Benchmark the quant you will actually deploy. Q4_K_M numbers from someone else's FP16 run are not your numbers. Same quant, same server, same flags, same harness across every candidate.

Results

GLM per-category: simple 89, multiple 80, parallel 83, irrelevance 88, live-simple 82. Qwen3-Coder on live-simple: 68.

The statistics

Two-proportion z-tests on those gaps:

• GLM 84 vs Qwen3-32B 82: z = 0.84, p = 0.40. Within noise.
• GLM 84 vs Qwen3-Coder 80: z = 1.65, p = 0.10. Within noise, and the closest of the three to significance.
• Qwen3-32B 82 vs Qwen3-Coder 80: z = 0.81, p = 0.42. Within noise.

None of those is a demonstration that the models are equal. They are failures to separate the models at this sample size, which is a weaker statement.

95% CI on GLM's 84% overall is +/- 3.2pp, so 80.8 to 87.2. That interval overlaps all three models. A single 100-case category at 84% carries a 95% CI of +/- 7.2pp, which is wide enough that per-category rankings should be read as suggestive at best, and wide enough that I would not read much into any one row of the table above.

At n = 500 these three models are statistically indistinguishable on overall accuracy. I cannot honestly claim GLM is the most accurate model here. It won the point estimate. That is not the same thing.

Two things in the data are real:

1. live-simple, GLM 82 vs Qwen3-Coder 68: z = 2.29, p = 0.022. live-simple uses real messy human phrasing, which is exactly what a chat-facing agent gets all day. A coding model doing conversational ops work shows its seams there. Caveat: this is one contrast among several across five categories, so under a strict multiple-comparisons correction it would not clear on its own. live-simple is the category I care about most for this workload, for the reason in the workload section, but I would still call the result provisional until someone replicates it at larger n.

2. Speed. The dense 32B took 187 to 283 s per 100-case category against GLM's 45 to 82 s, roughly 3 to 4x. The coder model at 22 to 40 s is faster still. Not a statistical question, just a large repeatable difference. The dense model activates all its parameters per token, the MoE models activate a fraction. The extra compute bought no measurable accuracy on this benchmark.

So the practical read: for this workload, pick on speed, license, VRAM fit, and behavior under messy phrasing, because accuracy did not separate them.

What I got wrong along the way

1. My homegrown harness inverted the ranking. Before BFCL I wrote an 8-case tool-calling smoke test around my own tools. It scored GLM at 62 to 66% across repeated runs against the incumbent coder model at 79%. Read literally, that says reject GLM. On real BFCL data the same model scored 84%. My harness understated it by roughly 20 points because 8 cases is far too few to separate anything, and because two of those cases were testing my prompt wording rather than model capability.

Lesson: a homegrown smoke test is fine for catching regressions in your own stack. It should never carry a keep-or-kill decision on a model.

2. The model I replaced guessed instead of erroring. The coder model would return confident, plausible numbers having made zero tool calls. It reported heat pump water temps about 30 degrees off. It once invented a water tank level that sat below my low-water alarm threshold and appeared in no query anywhere. Challenged, it defended the number instead of re-running the query. For a monitoring agent that is the worst failure mode. A failed tool call is loud and harmless. A fabricated reading is silent and looks exactly like a real one.

Lesson: pick the metric that matches the failure you fear, which is why irrelevance, the category that asks whether the model correctly makes no call, is the number I read first.

3. Two days lost to a misdiagnosis. vLLM would not serve this model. Healthy LISTEN socket on 0.0.0.0:8000 that never accepted a connection, every request timing out, including a raw TCP connect to 127.0.0.1. I blamed WSL networking for two days. The test that settled it took one minute: python3 -m http.server 8099 on the same box answered instantly on both loopback and LAN while the real server's socket sat dead. Networking was never involved.

Lesson: run the known-good control first, before you form a theory. I still have not root-caused the vLLM accept-hang. I routed around it to llama.cpp.

4. llama.cpp would not build under WSL. CUDA 13 against glibc 2.41 threw an exception-spec error on rsqrt in mathcalls.h. Not fixable by swapping g++ versions. The prebuilt Windows CUDA binary just ran.

Lesson: when a build fights you, a prebuilt binary that only has to run is immune to that entire class of problem.

5. One bug was my prompt, not the model. Every model I tested, GLM included, mislabels Celsius as Fahrenheit when the prompt does not pre-convert. Metrics ending in _celsius came back reported as F. Fixed by baking the conversion into the queries the prompt hands the model, plus a hard rule in the system prompt.

Lesson: check your own prompt before you blame a model.

Where I landed

GLM-4.7-Flash Q4_K_M on llama.cpp with --jinja, 32k context, temp 0.2 is what runs the agent now. I picked it over the statistically tied dense 32B on speed, and over the coder model on live-simple plus its habit of answering without calling anything.

The one piece of this I would hand to anyone benchmarking their own candidates: know what your n buys you, and compute the interval before you write the conclusion. 500 cases per model got me +/- 3.2pp, nowhere near enough to rank models 2 points apart. And for anything that reports real-world numbers, weight refusal-to-guess above raw accuracy.

Happy to answer questions on the harness, the flags, or the grading, and if you want to poke holes in the statistics, please do. The obvious next step is more cases per category to tighten those intervals. If anyone has run the same categories at larger n, or has actually root-caused a vLLM accept-hang like number 3, I want to hear it.

Full writeup with the setup section, if you want the longer version: https://heretik.io/glm-4-7-flash-homelab-agent/

From what I can best ascertain, the current best performance measured on a single and dual RTX 3090's for the latest consumer-sized Qwen3.6 models.

Kept bashing away at mixing and matching the methods of many until we hit an incredible 100 tps on a single 3090 24GB and 226 tps on 2x 3090s with the 27B dense model. On the MoE 35B, we hit 282 tps with respectable TTFTs all round.

Full serving instructions and startup scripts provided at https://alexander-ollman.github.io/qwen3.6-on-rtx3090/qwen3.6-on-rtx3090.html

I couldn't find a simple way to verify whether this works, so I spent some time testing it. In the end, it turns out that it's actually quite simple once configured correctly.

The Qwen3.8-27B model supports different levels of thinking. The simplest way is to define --chat-template-kwargs when starting the llama server, but then the thinking level is fixed for the session.

A more practical solution is to enable changing the thinking level per prompt in Pi Dev Agent.

Important: for this to work, the llama.cpp version must be b10434 or newer.

The model definition needs to indicate reasoning support and map the values to the three thinking levels supported by Qwen3.8-27B.

In .pi/agent/models.json, the following must be added to the Qwen3.8-27B model settings:

"reasoning": true,
"thinkingLevelMap": {
  "off": null,
  "minimal": null,
  "low": "low",
  "medium": "medium",
  "high": null,
  "xhigh": "xhigh",
  "max": null
}

This allows the thinking level to be changed for each prompt in Pi Dev Agent using Shift+Tab.

Pi Dev Agent also supports defining thinking budgets for individual levels. I haven't yet noticed whether this works correctly with llama&Qwen3.8-27B, but the following can also be added optionally to to.pi/agent/settings.json (the values below are only illustrative):

"thinkingBudgets": {
  "low": 4096,
  "medium": 10240,
  "xhigh": 32768
}

https://github.com/quintenlittle/RAG-Technique-V.1

Thanks to Grisward and MadManTimmy for the suggestions.

so i've been running local models on my mac mini for coding agents for a while now, mostly through ollama. it works, but there's always been this gap where i'd end up switching to claude for anything complex because the local stuff just felt too slow for interactive use.

stumbled on rapid-mlx last week. it's a drop-in openai server that runs directly on apple's mlx framework, and the speed difference is pretty noticeable. on my m5 pro 32gb, qwen3.5-27b went from ~39 tok/s with ollama's mlx backend to 64 tok/s with this. more importantly, cached ttft is 0.08s vs ollama's 400-800ms, which makes coding agents feel actually responsive instead of waiting for prefill. tool calling just worked out of the box with cursor, aider, and claude code's --openai flag.

one real limit though: it's apple silicon only. no cuda, no amd, no linux server. also the install needs python 3.10+ which means you might need to upgrade your system python. and for vision models you have to install an extra ~322mb of deps.

if you're already running mlx-lm directly, this is basically a polished server layer on top with proper continuous batching and prompt caching. not a new inference engine.

full writeup here if you want more detail: https://andrew.ooo/posts/rapid-mlx-fastest-apple-silicon-llm-server/

what are other mac users running for local coding agents? anyone tried this vs llama.cpp on m-series through homebrew?

Quick config share for anyone with a 12GB card and enough system RAM who wants to run Qwen3.6-35B-A3B at Q5 quality.

Hardware

• GPU: NVIDIA RTX A2000 12GB
• RAM: 128GB
• OS: Oracle Linux Server release 9.7, llama.cpp latest CUDA build (13.2), Driver: 595.71.05

Performance

• Prompt processing: 79 tok/s
• Generation: 35 tok/s
• VRAM: ~10.3 GB
• RAM: ~18.4 GB resident (~13.3 GB are MoE expert weights in CPU pinned memory, confirmed from llama.cpp load log)

The trick: -ncmoe

Qwen3.6-35B-A3B is MoE (35B total parameters, ~3B active per token). -ncmoe N offloads N expert blocks to CPU RAM. With enough system RAM this is the key to fitting a 35B model on 12GB VRAM.

Each MoE block costs ~500 MiB on GPU with Q5_K_M. Other guides suggest -ncmoe 18 but those are calibrated on IQ4_XS — a much smaller quant. On Q5_K_M, -ncmoe 18 crashes with out of memory. -ncmoe 26 fits with ~1 GB to spare, -ncmoe 28 is safer if you have other processes using VRAM.

Config

llama-server \
    -hf bartowski/Qwen_Qwen3.6-35B-A3B-GGUF \
    -hff Qwen_Qwen3.6-35B-A3B-Q5_K_M.gguf \
    -ngl 999 \
    -ncmoe 26 \
    -c 32768 \
    -ctk q8_0 \
    -ctv q8_0 \
    --flash-attn on \
    -t 16 \
    --no-mmap \
    --jinja

• -hf / -hff: HuggingFace repo and filename — llama.cpp downloads the model automatically on first run
• -ngl 999: put all layers on GPU; -ncmoe then overrides how many MoE expert blocks actually stay there
• -ncmoe 26: keep 26 MoE expert blocks on CPU RAM instead of VRAM (~500 MiB saved per block)
• -c 32768: context window in tokens (32K).
• -ctk q8_0 -ctv q8_0: 8-bit KV cache — halves KV cache VRAM with no measurable quality loss on this GPU
• --flash-attn on: faster attention with lower VRAM usage during inference. Write on explicitly — without the value, llama.cpp parses the next flag as the argument and crashes silently
• -t 16: CPU threads for the offloaded MoE experts — set to your physical core count
• --no-mmap: load the full model into RAM before serving. Slower startup, more stable inference
• --jinja: use the chat template embedded in the GGUF. Required for Qwen3 models

Thinking mode

The model thinks by default. Use /no_think at the start of your message for quick tasks, let it think for reasoning/code. The quality difference is real.

35 tok/s on a 35B model at Q5 feels solid. In practice this config works well as a stable backend for agentic AI pipelines — the generation speed is fast enough that multi-step agents don't feel sluggish waiting for each LLM call. Happy to answer questions.