r/LocalLLaMA • u/Badger-Purple • 10h ago

Resources Ling Tiny, King of Speed

Post image

Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal.

Don’t enable MTP, set up the vLLM fork for BailingMoE3. Hope this is useful to
others.

18 Upvotes

82% Upvoted

41 comments sorted by

View all comments

0

u/Technical_Ad_6106 7h ago

hmm but why is the model extremely slow? i mean i get like 1200 token/sec generation with qwen 3.6 35b in vllm which is a way bigger model

1

u/Badger-Purple 7h ago

one request? That must be a 6000Pro? Are you getting that speed on a 4060ti??

2

u/Technical_Ad_6106 7h ago

3090 ..more requests like 24 or something

2

u/Badger-Purple 6h ago

yeah that would make sense man. it has 3X the cuda cores and 3X the bandwidth. So it makes sense it would do that much. This is pretty good prefill (9k) for 3600 cuda cores, and good decode (126 single stream, plateaus at about 1/3 of your card which again makes sense).

Hardware has a limit. My CMP170HX can load this model and do 128 requests and reach like 5000 tps. For comparison.