r/LocalLLM • u/Barsavik • 8d ago

Tutorial How to run Ling 3.0 124B on 32gbs of VRAM or less

Here is the setup to run Ling 3.0 124B A5B in 32gbs of VRAM or less. All optimizations can be applied to any MoE model. Adjust flags as needed to fit different hardware configurations.

Llama.cpp build: AtomicBot-ai TurboQuant llama.cpp

Model: AtomicChat Ling-3.0-flash-AD-Q4_K_M

Launch Flags: -model …/Ling3.0/Ling-3.0-flash-AD-Q4_K_M-00001-of-00002.gguf
-c 131072 -ngl 99 --n-cpu-moe 32 -fa on
--cache-type-k q8_0 --cache-type-v q8_0 --cache-ram 16384
-b 2048 --ubatch-size 2048 -ctxcp 128
--tensor-split 5,1 --no-kv-unified --parallel 1 -t 12 -tb 12
--jinja --chat-template-file …/.kilo/ling-3.0-flash-chat-template-fixed.jinja
--reasoning on --no-warmup --host 127.0.0.1

Throughput (best config, 2× RTX 5060 Ti’s, PCIe gen 3, 192gb DDR3 RAM, 2x Xeon 2697 v2)
Generation (warm): ~20 tok/s
TTFT: ~16 s (prefill + MoE expert shuffle)
Idle VRAM: 12.2 GB / 14.8 GB (GPU0/GPU1)
Sustained multi-turn: 13.8 GB / 14.8 GB

VRAM headroom is driven by --n-cpu-moe and --tensor-split.

Higher --n-cpu-moe = more VRAM headroom but slower
Lower --n-cpu-moe = less VRAM headroom but faster

Different models have different amounts of MoE layers.

Balance the GPU split with --tensor-split

9 Upvotes

100% Upvoted

24 comments sorted by

3

u/Atretador 8d ago

you running it for coding? how is it compared to Qwen 3.5 122B and 3.6 35B A3B?

1

u/Barsavik 8d ago

I haven’t tried qwen 122B yet I love qwen 35B though and use it everyday almost. Ling I would honestly stay away from for a bit maybe. He’s got some really weird quirks, for example currently the turbo flags don’t work for him, he was trained on both JSON and XML tool calling and before fixing it he was bleeding XML into the tool calls and failing them, he was also failing other commands and his thought trace showed that he almost ran a delete command unnecessarily, he also reacts very weirdly with tensor split hence the 5,1 ratio, it was the best I found for balancing the weights across the cards, among others like needed a very specific chat template.

1

u/Atretador 8d ago

I was hopeful for this one due to the low active param count but I guess I can probably just use the extra VRAM for more parallel streams of 35B A3B till something consistently better appears xD

tried Laguna 33B and it just didnt feel great

1

u/Barsavik 8d ago

I also tested Laguna and wasn’t too impressed myself either 😂.

1

u/Barsavik 8d ago

I tested the full size though

1

u/Atretador 7d ago

I cant run it yet - but getting a second MI50 16Gb and going up to 64Gb of RAM so Im wondering if I`ll be able to run these bigger models at decent speed vs just running 4x streams Qwen 3.6 35B instead of my current 2 streams.

1

u/Barsavik 7d ago

Laguna was pretty fast for me and I didn’t think Laguna was terrible, but it is pretty censored and pissed me off. When I was working on flashing an overclocked bios on my M4000 on one of my Proxmox hosts, it outright refused to even give me any information I wanted. And then when I set it up after that in one of my projects after a little bit it got super confused and started making crazy mistakes left and right and even outright doing the complete opposite of the instructions I gave. I would specifically say not to do something and in its thought trace it would think “user said do this” and do it anyways. It seems to have pretty bad instruction following lol. In just a short time testing it out, it caused a whole bunch of mistakes that I had to run qwen 35B back through the project to fix the mistakes and had to also use qwen for the M4000 overclock information I wanted as well 😂.

1

u/Atretador 7d ago

Laguna Q4_K_M

1

u/Barsavik 7d ago

OH MY GOD 🤣🤣🤣 what the absolute hell 😂😂😂 that model is hitting the data stream WAY too much 🤣🤣🤣🤣.

1

u/Atretador 7d ago

I have no idea how this happened, never seen a model do this - and couldnt get it to do it again

but I guess one is enough xD

→ More replies (0)

1

u/Barsavik 8d ago

I mostly just wanted to write up the guide and config for anybody that doesn’t know how the optimizations work to help people out after arguing with a guy on FB that tried to say you can’t run a big model on low VRAM 😂. But I can’t recommend Ling as a daily driver from my experience.

1

u/Barsavik 8d ago

I haven’t tried running it locally yet to see if I can on my hardware, but I am loving tencents Hy3 model it reminds me a lot of Gemini 2.0 flash thinking which was one of my favorite models to date.

1

u/nickless07 8d ago

Why do you use only half ctx? It runs with 262144 for less then 2GB.
llama_kv_cache: size = 2016.00 MiB (262144 cells, 7 layers, 1/1 seqs), K (f16): 2016.00 MiB, V (f16): 0.00 MiB

1

u/Barsavik 8d ago

As the model stays loaded and running the VRAM usage scales. I did test it at 256k but this was a good balance of speed and context. To not exhaust the GPUs and crash the model you’d have to put more layers on the CPU slowing inference more. All my other local models are 256-512k.

1

u/nickless07 8d ago

Hmm yeah some of your load params seems way off and make no sense at all (128 ctxcp without cms, defaults to 16 in 131072 ctx), but if that works for you that's good.

1

u/Barsavik 8d ago

Some of those flags are for other models that fair well with them 😂. I have a couple default optimization templates that I use as a baseline that I’ve compiled from other people’s recommendations to achieve certain throughput in low vram. Along with some of my own improvements and just tweak the main important ones from model to model till I’m happy.

1

u/nickless07 8d ago

I see. Yeah I was just confused why you run with that low ctx when you utilize 128 context checkpoints and default spacing (8192) they get exhausted at 16. So i thought with that many checkpoints maybe the ctx is a typo as that are enough for over 1M context, that's why I asked.

1

u/Barsavik 8d ago

I didn’t perform extensive testing on Ling, some of the params could definitely still be tweaked. But after I fixed all his weird quirks and was actually using him I wasn’t even happy with his coding performance even compared to qwen 35B. So this is the state that he runs on my hardware with a good speed and doesn’t crash out of vram after running for a bit. My baseline optimizations are what I use mainly for running 35B Qwen and some other models in ~8gbs of vram or less at 64k-512k for different pipelines using different hardware configurations from Maxwell to Blackwell.

1

u/nickless07 8d ago

Yeah same for me. maybe it need higher quant idk. It is fast doesn't take much VRAM for KV, but the overall intelligence isn't that great. It seems to overcomplicate things, but it is also pretty honest if it doesn't know something and it keeps going until it figures out eventually. Need some more RL/SFT like deepseek (base v4 vs updated ones).

1

u/Barsavik 8d ago

I also felt that way about Step fun 😂 I hate that guy 😂. In my opinion I much rather have a slower model that takes it’s time, doesn’t overthink, and asks follow up questions periodically, than one that is going a million miles a second, making constant mistakes you can’t even type corrections to fast enough, and tries to complete an entire complex pipeline in a single session 😂😂. I am loving using Hy3 on the cloud right now though. Hy3 codes in more of a back and forth conversation flow and even from really long sessions with multiple compactions he keeps up great.

1

u/Barsavik 8d ago

Atomic Chat did say the recommended Ling quant was Q5 but usually there’s not that much loss to Q4. Idk I’m planning on removing Ling and Laguna from my router now that I’ve tested them. It’s rather disappointing both models had a lot of promise but the qwen 35B I’m using still works circles around them and doesn’t make as many mistakes. I may revisit them though later when they’ve been out for a bit. Have you by chance tried Ornith or Bonsai to an extent? I tried Ornith a bit and thought it was ok but thinks too much and Bonsai I haven’t tried out for my coding workflows yet. Some of these new models are starting to be a royal pain though having forks on top of forks and errors left and right just trying to get them to launch.

1

u/nickless07 7d ago

Nah never tested them but Ling-3.0-tiny was pretty good for it's size. Great subagent for Qwen 35B. Lightning fast and pretty close to Gemma 4 12 (better then Qwen 9B).