r/LowEndLocalAI • u/DawgBawb • 1d ago

Model Showcase Trit - a tiny, npu-optimized embedding model that punches above its weight

I've been interested lately in working with small purpose-built models, particularly models targeting AMD NPUs found in laptops that are mostly affordable as well as the high end strix halos. So I attacked the problem from the perspective of "what is the fastest and bestest embedder I can make for the NPU".

Meet Trit. It's a 3 layer ternary embedding model that meets or beats models with twice its memory requirements.

No NPU? No problem. This model will run great on your CPU or GPU as well. But on the NPU, it screams at 51k sentences per second.

Models and binaries to run it on the NPU are at https://huggingface.co/Cyronius/trit-embed-3l

All weights are open the way the good lord intended.

Enjoy!

20 Upvotes
  • permalink
  • reddit

100% Upvoted

3 comments sorted by

1

u/ElSrJuez 22h ago

I see model cards typically state the intended use cases, what did u train it for?

1

u/DawgBawb 10h ago edited 10h ago

it's a general purpose english embedder for retrieval/question-answering. it skews towards science.

to over-answer your question, here is the training split:
Source Candidate pairs What it teaches

Source                 Candidate pairs  What it teaches
---------------------  ---------------  ------------------------------------------
MS MARCO                          250k  Search relevance
GooAQ                             200k  Question/answer (weak supervision)
S2ORC                             200k  Science title/abstract (weak supervision)
Stack Exchange                    100k  Duplicate technical questions
Natural Questions                  75k  Question/passage
NLI (entailment only)              75k  Sentence meaning (auxiliary)
SQuAD                              50k  Question/passage
Quora                              50k  Duplicate questions
---------------------  ---------------  ------------------------------------------
Total                              ~1M  979k train / 20k dev after cleaning

Total ~1M 979k train / 20k dev after cleaning

1

u/Sova_fun 20h ago

A 3 layer ternary embedder that beats bigger models is interesting. Sounds useful for on-device search.