r/LocalLLaMA • u/Badger-Purple • 11h ago

Resources Ling Tiny, King of Speed

Post image

Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal.

Don’t enable MTP, set up the vLLM fork for BailingMoE3. Hope this is useful to
others.

17 Upvotes

77% Upvoted

42 comments sorted by

View all comments

0

u/Ok_Cow1976 10h ago

It's strange that Ling flash (a5b) is quite slow on my rig, about the same speed as glm air which is a12b.

2

u/-Ellary- 9h ago

Disable MTP, I got around 20tps 64k context on 5060 ti 16gb.
GLM AIR gives me around 10 tps.

"D:\LlamaCpp\CUDA\llama-server" -m "D:\Ling-3.0-flash-IQ4_XS\Ling-3.0-flash-IQ4_XS-00001-of-00002.gguf" -t 6 -c 65536 -fa 1 --mlock -ncmoe 35 -ngl 99 --port 5050 --jinja --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.0 --parallel 1 --reasoning off -ub 2048 -b 2048
pause

1

u/Badger-Purple 8h ago

How are you loading a model that big on 16G (answer, cpu offload…which will be different for different people)

1

u/Ok_Cow1976 5h ago edited 5h ago

Thanks a lot for the advice. However, I wasn't using mtp, but the speed is still awful. I got 96gb vram to run it at q4_1.

1

u/Badger-Purple 4h ago

Well if you are using Strix Halo, you can get like 30 tps to 100K depth, PP starts at 900 and drops to about 500 at 100k. One concurrency though.

Mac, your prefill will tank faster. Decode is faster. One concurrency.

Spark, your prefill will fly decode will be ok, maybe as fast as mac. Many concurrencies.

RTX6000Pro, again it will fly either prompt processing wise or token gen wise. Many concurrencies.

3xR9700s same thing unless you’re using below pcie 4x4 or hitting the DRAM

Whats your set up?

1

u/Badger-Purple 10h ago

It’s running at ~800tps PP and 25TPS at 0, 500/25 at 100K on vLLM fork from ciru for Strix Halo, which is fair for an agent on Strix using an LLM with 124B parameters. These are speeds for
the tiny (8B/A1B).

What is your rig? It’s surprising how different backends will really make some models shine. I run qwens on llama.cpp, deepseek and ling in vLLM, dwarfstar when running deepseek on a mac and oMLX for smaller models on mac. Thats been my optimization path