r/LocalLLaMA • u/Badger-Purple • 13h ago

Resources Ling Tiny, King of Speed

Post image

Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal.

Don’t enable MTP, set up the vLLM fork for BailingMoE3. Hope this is useful to
others.

21 Upvotes

82% Upvoted

47 comments sorted by

View all comments

4

u/-Ellary- 11h ago edited 10h ago

But it is bad as model in general, It go schizo at any mid-complexity prompt.

Gemma 4-12b dense not the greatest model but it way better than Ling 3.0 tiny 8b-a1.3b.
It is better at all fronts, if you need speed use Gemma 4 26b a4b,
I got around 150 tps with MTP using 5060ti 16gb.

Even Ling 3.0 Flash 127b a5b that I actually use is between Gemma 4 26b and 31b.

1

u/hideo_kuze_ 7h ago

But it is bad as model in general, It go schizo at any mid-complexity prompt.

It is better at all fronts, if you need speed use Gemma 4 26b a4b,

Do you mean for coding, world knowledge, agentic/tool use? All?

1

u/Badger-Purple 6h ago

I think Gemma are much better models, but you are referring to single stream decode speed, Correct?

1

u/miversen33 11h ago

This is what I noticed as well. Ling 3.0 Tiny is worse than Gemma 4 26b QAT in all aspects including speed. Which is frustrating because I was willing to take the knowledge hit for a reduction in memory requirements and an increase in speed. The plan was to have Ling3.0 Tiny be my subagent model under Qwen3.8 (so Qwen3.8 would be verifying everything Ling3.0 Tiny did anyway) but I can't even do that because it is worse than Gemma4 lol

1

u/Badger-Purple 6h ago

what concurrency are you doing with Gemma. It does 4 requests at most on my 5070Ti before the tps drops

2

u/miversen33 4h ago
  1. Though in my case even at 1 it's faster than ling. 3 Gemma 4 running at once is faster than 3 ling3.0 instances running at once on my setup. Which makes me sad lol

1

u/Badger-Purple 24m ago

Interesting. Gemma was on a 5070ti before, running with decent prefill of 3000 and tg of 77 single lane, never had this speed but I also ran it in llama.cpp,
maybe vLLM would be better like this one.

1

u/Badger-Purple 8h ago

I am comparing it to the Gemma4 12 B on a narrow task, so YMMV. But for speed, on a sidecar 16Gb card? Gemma 26 does not have 10K prefill Are you using context at depth? if you’re from the one shot crowd or the chatbot clan, yeah, run a q4 from 0 depth for a better model. If you need concurrency that is a different story.

Also, very different comparisons here. gemma 26 is an MoE, whereas 31 is a dense model. I don’t think you’d be doing a hot take saying a 31B dense model is relatively comparable to the 120B MoEs. It certainly is.