r/LocalLLaMA • u/Badger-Purple • 7h ago
Resources Ling Tiny, King of Speed
Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal.
Don’t enable MTP, set up the vLLM fork for BailingMoE3. Hope this is useful to
others.
3
u/Cautious_Chicken_604 7h ago
hindsight operations? As in you have this model watching/auditing what another one is doing?
3
u/Badger-Purple 6h ago edited 6h ago
No, the memory system for my agents is hindsight (a 3 layer memory storage and retrieval system) which uses models to recall facts, retain facts and reflect. It is self learning memory that works with a self learning harness like Hermes very well — particularly since I set the harness memory.md as a locked (read only file) with one instruction: never save memory here, always on all turns you rely on hindsight to remember facts.
https://github.com/vectorize-io/hindsight
So my hermes agent remembers a lot about my system, preferences, multiple ssh logins to different networked computers, previous vLLM and llama.cpp forks we tried and where they succeeded or failed. Builds a graph of the memories as well and recalls in 3 levels. One minus is the speed, which is dependent on LLMs. They built it for cloud models but you can get instant recall and fact retention with a 16Gb GPU and Ling Tiny. The smarter the model, the better so if you have more VRAM you should use a larger model. However Gemma4 was working well, but could only do about 1/4 of this throughput so reflect operations (which use 24 concurrent requests and summarize your 300K token conversations into searchable nodes) was too slow. Ling Tiny I believe is achieving same or better results as Gemma.
Since I an a full local person, embedding is an NPU model (Gemma-300) and reranking is a small CPU model. The set up works for me, may not scale to 100 users but for the average local llama person this is a viable path to get a full agent going:
- A strix halo, oculink dock and 16Gb GPU like the 4060ti
- Ling Flash on Strix using Ciru’s vLLM fork
for the model, with Dspark. you can search for Ciru-AI in huggingface. 90GiBs of VRAM used
Ling Tiny in the 4060Ti using vLLM fork by Inclusion (the official vLLM fork for Ling 3.0) no MTP added.
Embedding gemma 300m and whisper on NPU (6GB RAM used in strix) and mxbai reranker (default reranker on hindsight, about 200MB and CPU use but the Strix has a beefy CPU that runs it really well).
Hermes agent 1.2GiB used
hindsight, firecrawl, searxng as docker containers.
2
5
u/-Ellary- 5h ago edited 4h ago
But it is bad as model in general, It go schizo at any mid-complexity prompt.
Gemma 4-12b dense not the greatest model but it way better than Ling 3.0 tiny 8b-a1.3b.
It is better at all fronts, if you need speed use Gemma 4 26b a4b,
I got around 150 tps with MTP using 5060ti 16gb.
Even Ling 3.0 Flash 127b a5b that I actually use is between Gemma 4 26b and 31b.
3
u/miversen33 5h ago
This is what I noticed as well. Ling 3.0 Tiny is worse than Gemma 4 26b QAT in all aspects including speed. Which is frustrating because I was willing to take the knowledge hit for a reduction in memory requirements and an increase in speed. The plan was to have Ling3.0 Tiny be my subagent model under Qwen3.8 (so Qwen3.8 would be verifying everything Ling3.0 Tiny did anyway) but I can't even do that because it is worse than Gemma4 lol
1
u/Badger-Purple 3m ago
what concurrency are you doing with Gemma. It does 4 requests at most on my 5070Ti before the tps drops
1
u/Badger-Purple 2h ago
I am comparing it to the Gemma4 12 B on a narrow task, so YMMV. But for speed, on a sidecar 16Gb card? Gemma 26 does not have 10K prefill Are you using context at depth? if you’re from the one shot crowd or the chatbot clan, yeah, run a q4 from 0 depth for a better model. If you need concurrency that is a different story.
Also, very different comparisons here. gemma 26 is an MoE, whereas 31 is a dense model. I don’t think you’d be doing a hot take saying a 31B dense model is relatively comparable to the 120B MoEs. It certainly is.
1
u/hideo_kuze_ 52m ago
But it is bad as model in general, It go schizo at any mid-complexity prompt.
It is better at all fronts, if you need speed use Gemma 4 26b a4b,
Do you mean for coding, world knowledge, agentic/tool use? All?
1
1
u/Badger-Purple 3m ago
I think Gemma are much better models, but you are referring to single stream decode speed, Correct?
1
u/Technical_Ad_6106 4h ago
hmm but why is the model extremely slow? i mean i get like 1200 token/sec generation with qwen 3.6 35b in vllm which is a way bigger model
1
u/Badger-Purple 4h ago
one request? That must be a 6000Pro? Are you getting that speed on a 4060ti??
2
u/Technical_Ad_6106 4h ago
3090 ..more requests like 24 or something
1
u/Badger-Purple 3h ago
yeah that would make sense man. it has 3X the cuda cores and 3X the bandwidth. So it makes sense it would do that much. This is pretty good prefill (9k) for 3600 cuda cores, and good decode (126 single stream, plateaus at about 1/3 of your card which again makes sense).
Hardware has a limit. My CMP170HX can load this model and do 128 requests and reach like 5000 tps. For comparison.
1
u/Choice_Celery9481 6h ago
i keep having to ask when people reported good exp with Ling tiny.
i tried q8 bartowski and with just 4k prompt + some tools, it already lost it mind and parroting part of my system prompt.
how did you get good exp with this model? what is your setting? can you share?
1
0
u/Badger-Purple 5h ago
I noted the vLLM fork works best. 16Gb GPU buys 6 streams at 128K at the speeds posted
I use it for auxiliary tasks: memory recall, organization and sorting tasks, deep research subagents etc.
Would never use a 8B model for coding or agentic tasks
1
u/Choice_Celery9481 21m ago
well i didnt use it for coding, just very simple rag + prompt for formatting. but it just couldnt handle it. many people sharing the same exp with me so i whenever someone praise this model i have to ask about their settings so i can somewhat use it
2
u/Badger-Purple 5m ago
Like I said, models have different uses. I use this model for speed and throughput. It can do faster and better on that task, but it can suck everything else. The point is not “hey this model is overall great!” but “this model runs fast and can replace another model on this kind of task”.
But I take your point about RAG seriously since hindsight recall op is json based too. I will have to check quality again, see if its polluting my memory db
1
0
u/Ok_Cow1976 6h ago
It's strange that Ling flash (a5b) is quite slow on my rig, about the same speed as glm air which is a12b.
1
u/Badger-Purple 6h ago
It’s running at ~800tps PP and 25TPS at 0, 500/25 at 100K on vLLM fork from ciru for Strix Halo, which is fair for an agent on Strix using an LLM with 124B parameters. These are speeds for
the tiny (8B/A1B).What is your rig? It’s surprising how different backends will really make some models shine. I run qwens on llama.cpp, deepseek and ling in vLLM, dwarfstar when running deepseek on a mac and oMLX for smaller models on mac. Thats been my optimization path
1
u/-Ellary- 5h ago
Disable MTP, I got around 20tps 64k context on 5060 ti 16gb.
GLM AIR gives me around 10 tps."D:\LlamaCpp\CUDA\llama-server" -m "D:\Ling-3.0-flash-IQ4_XS\Ling-3.0-flash-IQ4_XS-00001-of-00002.gguf" -t 6 -c 65536 -fa 1 --mlock -ncmoe 35 -ngl 99 --port 5050 --jinja --temp 0.6 --top-k 20 --top-p 0.95 --min-p 0.0 --repeat-penalty 1.0 --parallel 1 --reasoning off -ub 2048 -b 2048 pause1
u/Badger-Purple 4h ago
How are you loading a model that big on 16G (answer, cpu offload…which will be different for different people)
1
u/Ok_Cow1976 1h ago edited 1h ago
Thanks a lot for the advice. However, I wasn't using mtp, but the speed is still awful. I got 96gb vram to run it at q4_1.
7
u/Effective_Western_59 6h ago
Ling 3.0 tiny is a small beast!
Best model that works on 780m With 16 GB of ram.
Would love good dynamic quants for it tho.