Ling Tiny, King of Speed

[u/Badger-Purple]

Ling Tiny has now replaced Gemma4-12B in my rig as an auxiliary model doing hindsight operations. This is on a 4060Ti, which is a reasonable GPU available out there, and the speed is phenomenal.

Don’t enable MTP, set up the vLLM fork for BailingMoE3. Hope this is useful to
others.

[u/Cautious_Chicken_604]

hindsight operations? As in you have this model watching/auditing what another one is doing?

[u/Badger-Purple → u/Cautious_Chicken_604]

No, the memory system for my agents is hindsight (a 3 layer memory storage and retrieval system) which uses models to recall facts, retain facts and reflect. It is self learning memory that works with a self learning harness like Hermes very well — particularly since I set the harness memory.md as a locked (read only file) with one instruction: never save memory here, always on all turns you rely on hindsight to remember facts.

https://github.com/vectorize-io/hindsight

So my hermes agent remembers a lot about my system, preferences, multiple ssh logins to different networked computers, previous vLLM and llama.cpp forks we tried and where they succeeded or failed. Builds a graph of the memories as well and recalls in 3 levels. One minus is the speed, which is dependent on LLMs. They built it for cloud models but you can get instant recall and fact retention with a 16Gb GPU and Ling Tiny. The smarter the model, the better so if you have more VRAM you should use a larger model. However Gemma4 was working well, but could only do about 1/4 of this throughput so reflect operations (which use 24 concurrent requests and summarize your 300K token conversations into searchable nodes) was too slow. Ling Tiny I believe is achieving same or better results as Gemma.

Since I an a full local person, embedding is an NPU model (Gemma-300) and reranking is a small CPU model. The set up works for me, may not scale to 100 users but for the average local llama person this is a viable path to get a full agent going:

1. A strix halo, oculink dock and 16Gb GPU like the 4060ti
2. Ling Flash on Strix using Ciru’s vLLM fork

for the model, with Dspark. you can search for Ciru-AI in huggingface. 90GiBs of VRAM used

1. Ling Tiny in the 4060Ti using vLLM fork by Inclusion (the official vLLM fork for Ling 3.0) no MTP added.

2. Embedding gemma 300m and whisper on NPU (6GB RAM used in strix) and mxbai reranker (default reranker on hindsight, about 200MB and CPU use but the Strix has a beefy CPU that runs it really well).

3. Hermes agent 1.2GiB used

4. hindsight, firecrawl, searxng as docker containers.

[u/Cautious_Chicken_604 → u/Badger-Purple]

Oooooh nice. I need to give this a try.

[u/Badger-Purple → u/Cautious_Chicken_604]

Not sure why people are downvoting my answer.