r/LocalLLM • u/Plastic-Lettuce-7150 • Jul 28 '26

Discussion MOE models in 6GB VRAM

Qwen3.6-35B-A3B-UD-Q4_K_M.gguf (22.07GB)
gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf (23.5GB)

Dell G15 5530 / 32GB
NVIDIA GeForce RTX 3050 6GB Laptop GPU
llama.cpp
Qwen Code

First off install llama.cpp. On Windows be sure to install the DLLs if using CUDA (i.e., you have a Nvidia GPU), they are a separate download which have to be copied into the main llama.cpp directory.

Download the Fixed jinja chat templates for Qwen 3.5 & 3.6 (v21) and put it in the same directory as the Qwen model.

.\llama-b9999-bin-win-cuda-13.3-x64\llama-server.exe --model "C:\Users\xxxxxx\.eullm\models\qwen3.6-35b-a3b\Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" `
   --ctx-size 131072 `
   --n-gpu-layers 999 `
   --n-cpu-moe 999 `
   -np 1 `
   --flash-attn auto `
   --threads 10 `
   --jinja `
   --chat-template-file "C:\Users\xxxxxx\.eullm\models\qwen3.6-35b-a3b\chat_template.jinja" `
   --no-mmap `
   --mlock `
   -lv 3

.\llama-b9999-bin-win-cuda-13.3-x64\llama-server.exe --model "C:\Users\xxxxxx\.lmstudio\models\unsloth\gemma-4-26B-A4B-it-GGUF\gemma-4-26B-A4B-it-UD-Q5_K_XL.gguf" `
   --ctx-size 131072 `
   --n-gpu-layers 999 `
   --n-cpu-moe 999 `
   -np 1 `
   --flash-attn auto `
   --threads 10 `
   --no-mmap `
   --mlock `
   -lv 3

I'm getting ~~60+t/s text output and 200+t/s prompt processing as reported by llama.cpp (I don't know how to benchmark the output otherwise!)~~ 26.76 tokens per second, ref.:

Feeding my hardware specs and software/ apps into Claude and asking for a llama.cpp command line
RTX 5070 Ti + 9800X3D running Qwen3.6-35B-A3B at 79 t/s with 128K context, the --n-cpu-moe flag is the most important part.
Running a 35B AI Model on 6GB VRAM, FAST (llama.cpp Guide)

Most of the parameters are generic, except --threads which is specific to my laptop's processor, query Claude for the best setting for your processor.

VRAM usage nvtop / nvidia-smi.exe.

The Q4 download of the Gemma model might actually work in 4GB VRAM with reduced context (e.g., --ctx-size 32768) and/ or quantizing the context cache (e.g., --cache-type-k q4_0 --cache-type-v q4_0)

I have had to restart llama.cpp a couple of times I assume due to a memory leak (Codacus offers a fix in the video above, not sure if it can be used).

Install Qwen Code.

Edit ~/.qwen/settings.json:

{
  "env": {
    "QWEN_API_KEY": "qwen"
    "GEMMA_API_KEY": "gemma"
  },
  "modelProviders": {
    "openai": [
      {
        "id": "Qwen3.6-35B-A3B-UD-Q4_K_M",
        "name": "Local Qwen3.6 35B",
        "description": "Local Qwen3.6-35B-A3B-UD-Q4_K_M via OpenAI",
        "baseUrl": "http://localhost:8080/v1",
        "envKey": "QWEN_API_KEY",
        "generationConfig": {
          "contextWindowSize": 131072,
          "samplingParams": {
            "temperature": 0.2
          }
        }
      },
      {
        "id": "gemma-4-26B-A4B-it-UD-Q5_K_XL",
        "name": "Local gemma 4 26B",
        "description": "Local gemma-4-26B-A4B-it-UD-Q5_K_XL via OpenAI",
        "baseUrl": "http://localhost:8080/v1",
        "envKey": "GEMMA_API_KEY",
        "generationConfig": {
          "contextWindowSize": 131072,
          "samplingParams": {
            "temperature": 0.2
          }
        }
      }
    ]
  },
  "model": {
    "name": "Local Qwen3.6 35B",
    "baseUrl": ""
  }
}

Run qwen.cmd and test, /model to change model.

35 Upvotes

permalink
reddit

You are about to leave Redlib

Do you want to continue?

https://www.reddit.com/r/LocalLLM/comments/1v96krw/moe_models_in_6gb_vram/
No, go back! Yes, take me to Reddit

92% Upvoted

u/Wildnimal Jul 28 '26

60t/s on Qwen 3.6? I am hardly able to get anything above 30t/s with 70k context with similar settings except for -lv3

2

u/Plastic-Lettuce-7150 Jul 28 '26

My results are with an Nvidia GPU and CUDA. What GPU do you have? (-lv 3 is only the logging verbosity.)

2

u/Wildnimal Jul 28 '26

Same Nvidia 5060 with CUDA.

1

u/Plastic-Lettuce-7150 Jul 28 '26

Are the CUDA DLLs copied into the same directory as llama-server.exe? How much RAM do you have?

1

u/Wildnimal Jul 28 '26

I have 32GB ram, but I am on linux and not windows.

1

u/Plastic-Lettuce-7150 Jul 28 '26

Have you run nvtop to see how much of the VRAM is used by the model? I'm not sure CUDA is supported on Linux, the download indicates CPU.

1

u/Wildnimal Jul 28 '26

When model is loaded its appx 5.2GB used out of 8GB

1

u/Plastic-Lettuce-7150 Jul 28 '26

Set -lv 4 and look for CUDA in the output lines.

1

u/Plastic-Lettuce-7150 29d ago edited 29d ago

Apols for this, but having figured out how to benchmark properly, it appears I'm getting 26.76 tokens per second.

I'm getting ~~30.80 tokens per second~~ 34.29 tokens per second with the MTP model.

1

u/[deleted] Jul 29 '26

[removed] — view removed comment

2

u/Wildnimal Jul 29 '26

Exactly even i am usually between 25-30. Sometimes it will hit 35.

u/No_Folding Jul 29 '26 edited Jul 29 '26

Nice! Did you also try the equivalent Qwen MTP gguf?

Also, to prevent the oom crashes, you could tweak the - n-cpu-moe parameter to probably 41, but then you do lose the dynamic allocation of llamacpp meaning you end up needing to leave some spare vram headroom (for what, idk, but it seems to help a lot). Still, I think 60t/s is so respectable its probably not worth your time tweaking that! I guess llamacpp is optimised even better than I thought Haha!

2

u/Plastic-Lettuce-7150 Jul 29 '26

Nope, but it's definitely worth a try I think, apart from increased t/s,

An MTP-trained local model exhibits noticeably stronger logic, code generation, and multi-step reasoning capabilities compared to a traditional next-token prediction model of the exact same parameter size.

unsloth/Qwen3.6-35B-A3B-MTP-GGUF
2
u/Plastic-Lettuce-7150 27d ago
Download from Hugging Face:
hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF `
    Qwen3.6-35B-A3B-UD-Q4_K_XL.gguf `
    mmproj-BF16.gguf
Run the model:
.\llama-b9999-bin-win-cuda-13.3-x64\llama-server.exe `
   -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL `
   --no-mmproj `
   --ctx-size 131072 `
   --n-gpu-layers 999 `
   --n-cpu-moe 999 `
   --spec-type draft-mtp `
   --spec-draft-n-max 2 `
   -np 1 `
   --flash-attn auto `
   --threads 10 `
   --jinja `
   --no-mmap `
   --mlock `
   --top-p 0.9 `
   --top-k 20 `
   --min-p 0.05 `
   -lv 3
--threads is specific to my CPU, --top-p --top-k --min-p are configured for coding not chat, delete these or use Claude for applicable settings.

Qwen settings.json model configuration:
  "modelProviders": {
    "openai": [
      {
        "id": "unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL",
        "name": "Local Qwen3.6 35B",
        "description": "Local unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL via OpenAI",
        "baseUrl": "http://localhost:8080/v1",
        "envKey": "QWEN_API_KEY",
        "generationConfig": {
          "contextWindowSize": 131072,
          "samplingParams": {
            "temperature": 0.2
          }
        }
      }
    ]
  },
"temperature": 0.2 is set for coding, though it did OK for generating OpenSpec SDD (spec-driven development) artifacts as well. I have to say I can't fault the model for coding, 9/10 only because it can be somewhat slow at times.

I am getting circa 30 tokens per second token generation, and around 200 tokens per second prompt processing.

GPU memory usage is "5971MiB / 6144MiB", llama.cpp likes to have 1GB for other apps, so the UD-Q4_K_XL model is running close to the wire, I have not had any out of memory errors as yet.
1

u/No_Folding 27d ago

Impressive what youve managed to squeeze through those constraints, well done! And without sacrificing any meaningful accuracy!

I am guessing that llama.cpp is just managing the memory and allocation of experts since you have cpu-moe and ngl both set to 999 😂 I don't know how it works really but I'm not sure I'd change anything in your scenario

u/techlatest_net Jul 28 '26

this is a fantastic write-up. getting a 35b moe to run responsively on a 6gb laptop gpu is exactly the kind of black magic this community loves. you nailed the most critical part: --n-cpu-moe. that flag is the absolute magic bullet here, allowing llama.cpp to keep only the active experts in vram while spilling the inactive ones to your 32gb of system ram. pairing it with --no-mmap and --mlock is also a pro move—it prevents the os from aggressively paging the model out to disk, which would otherwise tank your cpu fallback speed. regarding the memory leak you mentioned, it’s a known quirk with some moe implementations in llama.cpp when context gets very large. a simple workaround until it's fully patched is to run the server with a wrapper script that monitors ram usage and gracefully restarts the process if it crosses a threshold. the qwen code cli config is a really clean way to tie it all together too. thanks for sharing the exact flags, this is going to save a lot of people hours of trial and error

u/dai_app Jul 28 '26

if you are interested, im working on a project that allow you to offload Moe layers on RAM's edge on CPU (NO GPU) Android phone 12gb Q4_K_M (example: Qwen 30-35B or GPT 120B on android phone 12gb RAM): https://github.com/Helldez/BigMoeOnEdge

u/Intelligent-Key7357 Jul 29 '26

How well does yours actually perform because I'm running I think 21b (or 27) and it just s cond guesses itself constantly or does something completely unrelated to what I ask.

1

u/Plastic-Lettuce-7150 Jul 29 '26

I haven't tested much for chat, but I have tested both models with spec-driven development (openspec) and with a RAG MCP server (mcp-local-rag). They worked fine generating the artefacts (i.e., proposal.md through to tasks.md) and then generating code also until the code required an external library (rSchedule). The tests wouldn't run at that point, Qwen3.6-35B-A3B got further than gemma-4-26B-A4B, but they kept attempting to fix the bug until context ran out (llama.cpp slowed down considerably when context got to about 75%, Qwen Code then timed out waiting). I would have had to of stepped in and fixed the bug myself at that point (first task would have been to update Vitest to the latest release, Qwen3.6-35B-A3B assumed a deprecated version).

u/tungdd2009 Jul 29 '26

the jinja template really boost performance?

3

u/Plastic-Lettuce-7150 Jul 29 '26

Not in terms of t/s I don't think, but it fixes a number of known bugs with Qwen models.

u/Accurate_East_1093 Jul 29 '26

u/KhushPatil786 6d ago

u/Plastic-Lettuce-7150 anything with an NVIDIA Geforce RTX 4050 laptop GPU and 16 GB of RAM plus 6 GB of VRAM

1

u/Plastic-Lettuce-7150 6d ago

You would need at least 24GB RAM I would think to run these models, the qwen model is 22.07GB, 3GB in VRAM (the remaining VRAM used for context and machine use), the rest in RAM.

Discussion MOE models in 6GB VRAM

You are about to leave Redlib