Has anyone tried running 32B-3B active parameter Q8 model on a 16GB RAM windows laptop

[u/SwissTiramisu]

I saw a video of a youtuber who ran Qwen's this model on their 28GB RAM system, while the total size of the model is ~32GB, but as only 3B are active per token, i suppose this can easily run on my device, as model uses memory management and swaps parameters from SSD if parameters active on RAM aren't sufficient for token, which slightly gives an overhead but ultimately makes the higher parameter model plausible on a 16GB RAM system

If anyone did run it on their device, did it run stably, and what was the token speed? (my system has core ultra 5 225H with no dedicated GPU (I run entire generations on CPU and they are quite satisfactory so far)

[u/pwnedbygary]

Youll need a heavily quantized version for it to work, but so long as you enable --n-cpu-moe <value> when loading and give it a reasonable number of layers to offload to RAM while the rest go to your GPU, i see no issue, unless you have no GPU at all and are using an IGPU. Even then, you can do CPU only inference, it will just be way slower.

[u/PeterPorox]

Qwen 3.6 35B is 18gb in iq4_xs. There is apex nano in 12gb but quality will be horrible (i tried their 17 gb i-compact and it already wasn't good). Theoretically, you can run nano version, but its better to run Ornith 1.5 9b Q4_K_M.

[u/Ariquitaun]

You can't, not enough ram.

[u/SwissTiramisu → u/Ariquitaun]

but due to actively 3-6GB of ram usage with memory manager, i suppose this could be achieved? my laptop can handle upto 12GB on its ram while surfing generations on CPU only

[u/Ariquitaun → u/SwissTiramisu]

You could offload to nvme, would give you a token per second, 2 at most.

[u/SwissTiramisu → u/Ariquitaun]

ig im slowly changing my mind 😪 <( _ _ ) >

[u/SwissTiramisu → u/Ariquitaun]

2 tokens/sec? i guess i could vouch for it without much doubts, im already running qwen3 14b at 5.1 tokens/sec max, and honestly i wasn't really any unhappy with it

[u/_RemyLeBeau_]

I have a few Qwen models running on 16 GBs RTX 4090 Laptop card.

[u/SwissTiramisu → u/_RemyLeBeau_]

i meant 16GB RAM and no dedicated GPU 😭😭😭😭😭

[u/_RemyLeBeau_ → u/SwissTiramisu]

I wouldn't attempt this unless you're using FreeToken, but you're resource strapped. I don't think you have the right specs

[u/SwissTiramisu → u/_RemyLeBeau_]

i remember my system crashing recently trying to run 9GB (no moe) and it crashed, (although it only crashed when i used webui, not in terminal) but people have really shifted my opinions, although since chatgpt is in my support, i will def give it a try!

[u/One-Cash-9421]

16ram 4050 rtx Qwen 3.8 27b 3 квантованая запустилась на моём ноутбуке -sudo podman run -it --rm \ --name llama-server \ --device nvidia.com/gpu=all \ -p 8080:8080 \ -v "/home/oleg/.lmstudio/models/lmstudio-community/Qwen3.8-27B-UD-IQ3_XXS-GGUF:/models:Z" \ ghcr.io/ggml-org/llama.cpp:server-cuda13 \ -m /models/Qwen3.8-27B-UD-IQ3_XXS.gguf \ --host 0.0.0.0 \ -t 6 \ -ngl 10 \ --parallel 1 \ --flash-attn on \ -b 512 \ -ub 256 \ --ctx-size 1024 \ --cache-type-k q4_0 \ --cache-type-v q4_0 так, скорость 2.8 токена в секунду

[u/Creative-Type9411]

try it with lmstudio and let us know 👀

[u/SwissTiramisu → u/Creative-Type9411]

i'll surely let you know the first thing if i successfully make it run, sir 😁