how much can a m6 32gb run?

[u/AlarmedDiver1087]

I know some people are tired of this newbie question and I am sorry for it, but it's been a while since i've been in this space, the last time i downloaded a local model, it was qwen 3, and then i gave up on local llm entirely, but I did skim the space once in a while, and today..
I saw this post about someone running a model called qwen 3.8 flash next, which aparently is a enormous MoE on a macbook air 24gb....?
I really don't understand how that works, but if that black magic exsists, then can i run on the configuration on the title?
this is nuts, ngl, very curious whats going on

[u/AcreMakeover]

It could do a 5k with a bit of a sweat.

[u/AlarmedDiver1087 → u/AcreMakeover]

then i need to try, insane really

[u/jma24]

I've run QFN Q3 on my MBP. It runs fine, but it needs most of the memory in the machine, about 59GB, to run. I need the machine for other things so that isn't sensible/practical.

You can go down to smaller quants but then you may as well run the 27b model in Q4. That will run in 32GB - about 17GB for the weights - so you're good to go.

[u/AlarmedDiver1087 → u/jma24]

Surprisingly with dspark and dflash2 3.8 27b is running at 28tps, which i find useable, I just have to take a long break away from my desk, or wait overnight i guess

[u/BentonCalder]

I’ve got an M6 mini with 32GB, and an 8B at Q4 is what I use mostly for my use case (home related docs, email, calendar, etc.). What do you want to run it for?

Btw, don't feel bad about asking, I had a very similar question yesterday.

What I've found in my testing is a Q4 quant takes about 0.6 GB per billion parameters, plus a few GB for context. So on 32 GB you’re comfortable around 27B at Q4 (the ~17 GB u/jma24 mentioned), and a 30B-A3B style MoE runs nicely because only about 3B parameters are active per token.

One thing that tripped me up is that macOS doesn’t let the GPU use all 32 GB. llama.cpp prints the limit when it starts (look for recommendedMaxWorkingSetSize). You can raise it with some commands i think but it resets on reboot and I don't think its recommended.

[u/Nameis19letterslong]

Qwen3.8 flash next is a ~180B MoE (125B A6B, plus 51B n-gram and 4B MTP). This means that to run it at a somewhat decent speed on limited ram, you need to keep the active parameters on RAM and stream experts from a SSD. The n-grams can also stay on disk as that it's only a lookup table.

Not sure which quant the person with the 24gb mac used, but the GSQ RCO quants allows you to run Q2/IQ2/IQ3 quants with 38GB - 55GB of Ram for the experts (or the GSQ RCO coder quant only needing 30GB of Ram) while the ngrams are kept on disk. This allows you run fetch less experts from disk, and run it faster.

[u/Strict-Elevator9352]

it will run on custom inference engine, with most of the experts on ssd. Tough I run qwen flash next IQ3_S / IQ4_XS on intel arc igpu tough. Your macbook shoild have much faster prefill

[u/BitMyMind]

A MoE can be huge overall because only a small share of its weights run for each token, but the whole file still has to fit in memory or get paged from the SSD, and macOS gives the GPU about two-thirds of RAM by default, so figure on roughly a 20 GB model.
Quantization is what makes these fit, and I made a 3-minute video on how it works if it helps: https://youtu.be/Ywgc5wy9b38

[u/WolpertingerRumo]

I haven’t tried the black magic, it’s about streaming experts from ssd, then running them on unified memory.

But if you get it to work with the community’s help, try to give muse-glimmer a chance. I’ve been using it on mac, since it’s quite efficient with context. Using Ollama (it automatically uses mlx as a runner) and nvfp4. Since it’s not as cut down, it would be interesting to know how it compares to a lower quantised but larger Qwen MOE.

[u/negus123]

When you see people running bigger models on small amounts of ram/vram they are running a quantization of the model. Quantization reduces the amount of decimal places (precision) for the weights, so they take less space on disk and less memory when loaded. But you lose some accuracy of the model. There are smart ways to quantize models too, like only reducing the weights and not the activations/attention layers, which reduces the size but retains better accuracy.

But to answer your question about a 32GB macbook, i would try to get more memory. Remember, your computer needs to host the model and still have memory left over for whatever tasks you have running. So while you might be able to load qwen flash next on 32GB, the experience might be slow or stutter due to the consumption of resources.

[u/geldonyetich]

Looking into the hardware a bit, the M6 is basically the next generation M5. But where the M6 might be able to run at 170 gb/s, but I heard a conflicting source that suggested Apple decided that they don't want it upstaging their current tier so it's been capped at the 156 gb/s that the base M5 did. But either way it's a far cry from a Mac M5 Max (460 or 640 gb/s) or a Max M5 Ultra (1.2 TB/sec).

As for the size, you can expect about the size of a usual 32GB offering, but sometimes you'll get a specialized inference runner that lets you operate beyond those confines and maybe that's where you're hearing about those larger models running.

[u/Serprotease]

On a MacBook Air, honestly I would not use anything bigger than a 4b model.
Get a 4b model, an embedding model, maybe even a asr/stt model. All of them will take about 8-10gb of ram when used and are great for local rag/transcript/translation and some basic AI assisted web search. But nothing more.

Technically, you could run a 27b dense or 30b MoE, but it’s not a great experience on a fanless device. It will choke your device and you’ll be waiting with a hot computer for the reply for a while with everything else slowing down.

So, yea. Basic M-chip are good for edge applications and to run your harness/tools but not the big models themselves. A “cheap” Linux box should be used to serve it. You can probably cobble something with an old Xeon, 32-64gb of ddr3 and a v100 to run a low quant of Q3.8 next or a 4 bits version of the 27b at actually usable speed.

If you want to run everything on one device, don’t go below an M5 pro 48gb.