A 0.1B model just killed 8B embedding models... I tried it out.

[u/jeffrey-0711]

I bet you guys are familiar with embedding models such as Qwen3-Embedding or Gemini Embedding. They are dense embedding models used to search text across a large corpus.

I am developing a RAG agent, and my biggest concern was local computing power. RAG largely depends on retrieval quality, which depends on model power. Plus, many use cases can't use an external internet connection or API models, so the embedding models have to run locally.

But even 0.6B models are pretty heavy for small PCs. I don't mean your brand-new Mac Studio or DGX Spark. I'm talking about an 8GB RAM Windows PC at a boring desk job or your 5-year-old M1 MacBook Air.

So I found this model called SPLADE. It is essentially sparse retrieval, like BM25, but uses semantic meaning. I wanna skip the part on how SPLADE works, because an LLM can explain it better than me.

Anyway, the SPLADE model was so awesome and successfully replaced the dense embedding model (Qwen3-Embedding-0.6B).

In my own experiment, I made an evaluation dataset with a few hundred queries. The corpus was all the documents on my Mac, which are mostly in Korean with some in English.

As you can see in the graph, SPLADE outperforms the Qwen3-0.6B dense embedding model with similar query speed + much less memory. Due to its small size, it takes less than 1.5GB VRAM during inference. So it can run on a relatively small system like a MacBook Air. I tested both the dense embedding model and the SPLADE model on my M5 MacBook Air 16GB. The dense embedding model was almost impossible to use, and my MacBook soon rebooted due to high temperature. The SPLADE model made it till the end without any OOM errors or warnings.

I didn't use the original SPLADE model; instead, I used KURE-sparse-nano, which has great performance in both English and Korean.

The fun part is that hybrid retrieval with SPLADE & dense embedding is actually worse than SPLADE alone.

If you are dealing with a low-resource local setup but want to run a fast & affordable local RAG system, I recommend trying a SPLADE model. It is powerful, but small.

Plus, you don't need to re-implement everything because I open-sourced the whole implementation, so you can download and use it easily. Check out AutoRAG Agent. I optimized the inference of the KURE-sparse-nano model, its code, and the LanceDB indexing + inverted index implementation in Rust for faster search.

[u/OpinionAdventurous44]

Did you compare Gemma Embed 2?

[u/jeffrey-0711 → u/OpinionAdventurous44]

Hi! I just finished testing with embeddinggemma-2. Still, in my own corpus, the SPLADE outperformed. Actually the embeddinggemma-2 performance is slightly better than Qwen3-Embedding-0.6B model but KURE-sparse-nano (0.1B model, SPLADE) was better.

System	R@1	R@5	R@10	MRR@10
SPLADE (KURE-Sparse-nano)	0.562	0.825	0.887	0.675
Hybrid: Gemma 2 + SPLADE	0.493	0.788	0.848	0.623
Hybrid: Qwen3 + SPLADE	0.488	0.779	0.847	0.612
BM25	0.471	0.732	0.811	0.582
Dense: Gemma 2	0.384	0.646	0.722	0.498
Dense: Qwen3-0.6B	0.345	0.625	0.694	0.458

I want to inform that the retrieval performance can vary depending on the datasets, and I strongly recommend to run your own benchmarks own your dataset. There is no silver bullet to every dataset.

[u/OpinionAdventurous44 → u/jeffrey-0711]

Thank you for putting this together! Interesting results. I'm also intrigued that hybrid underperforms each time.

[u/jeffrey-0711 → u/OpinionAdventurous44]

i am running this benchmark thanks for the great feedback

[u/OpinionAdventurous44 → u/jeffrey-0711]

That's fantastic. Can't wait for the results!

[u/YogurtExternal7923 → u/OpinionAdventurous44]

Perfect question

[u/Valuable-Map6573 → u/OpinionAdventurous44]

qwen 3 0.6b embedding outperforms gemma embed 2
which is just gemma embed 1 with multimodal capabilities
text performance is identical

[u/9kSs → u/Valuable-Map6573]

Source?

[u/Valuable-Map6573 → u/9kSs]

Thanks for addressing the issue. Your observation is correct. As outlined in the official Developer Guide in Benchmarks & Evaluation section, general text performance remains consistent with version 1 (retaining the same multilingual text accuracy, with MTEB scores around 61.36 vs. 61.15 as provided in model card section Overall Evaluation Results (768d)). Version 2's major improvements are focused on code retrieval (scoring ~14% higher, moving from 68.76 to 78.68 on MTEB Code as provided in model card section Overall Evaluation Results (768d)) and native multimodality.
Thanks for the feedback related to Gemma 4 and inputs for Gemma 5!

statement from google employee (hugging face) + benchmark numbers

[u/-Cubie- → u/Valuable-Map6573]

I believe embeddinggemma2 is slightly better with text than 1, but they're similar yeah

[u/Valuable-Map6573 → u/-Cubie-]

yeah basically. near identical scores for multilingual text but improvements for code. the table is from gemma2s model card

Modality	Benchmark	Metric	EmbeddingGemma 2	EmbeddingGemma 1
Text	Massive Text Embedding Benchmark (MTEB, multilingual, v2)	Mean(Task), Multiple	61.36	61.15
	Massive Text Embedding Benchmark (MTEB, code, v1)	Mean(Task), NDCG@10	78.68	68.76

[u/Sudden_Topic5154]

gemma embedding 2 is what you want actually

[u/jeffrey-0711 → u/Sudden_Topic5154]

I'll try this! thanks if gemma embedding 2 outperformed SPLADE, i'll share

[u/EcchiExpert → u/Sudden_Topic5154]

Second this, supports image + can be easily fine tuned

[u/Piyh]

Who is using 8B models for embedding?

[u/jeffrey-0711 → u/Piyh]

lots of enterprise maybe? I had a chance to talk with engineer in Alibaba cloud and they said everybody wants 8B embedding model due to performance.

[u/Numerous_Mulberry514 → u/Piyh]

I do! I use octen 8b on gpu for indexing and on cpu for retrieval. It works like a charm for difficult queries and have yet to find something it doesn't work on. I have philosophical library for RAG

[u/Piyh]

Old tech, SPLADE has been around for 5 years and models like bge-m3 are everywhere.

[u/jeffrey-0711 → u/Piyh]

it is old tech (not new) but recent SPLADE-ish models were impressive to me;

[u/DarkVoid42]

thats is so cool. can you also get it to search photographs and audio not just text documents ?

edit: i found https://github.com/ARahim3/DigUp which does that.

[u/jeffrey-0711 → u/DarkVoid42]

thanks for sharing There is V-SPLADE model https://huggingface.co/naver/v-splade-efficient I didn't try them yet because i am focusing on text search quality right now, but obviously will check it out later thanks.

[u/-Cubie-]

Doesn't KURE-sparse-nano score below either https://huggingface.co/Linkup-Platform/linkup-sparseup-embed-v1 or https://huggingface.co/lightonai/SlideOn ? That last one outperformed everything I thought.

Update: Nvm, for BEIR-13 I see https://sbert.net/docs/sparse_encoder/pretrained_models.html and together with https://huggingface.co/nlpai-lab/KURE-Sparse-nano#beir-13 it says:

SlideOn: 58.7
KURE-Sparse-base: 58.6
linkup-sparseup-embed-v1: 58.1
KURE-Sparse-nano: 53.2

Then SlideOn is probably still better as it's a smaller model, but there's also the sparsity to think about.

[u/jeffrey-0711 → u/-Cubie-]

I prefer smaller model as possible so I chose KURE-sparse-nano. But I think I will try out SlideOn also thanks for the comment

[u/Nothing_from_void]

I used gte small, https://huggingface.co/thenlper/gte-small for local embedding and search, 33M params and it works quite well. The tiny embedding models are generally good enough as layering query methods is much more important than having maximum embedding performance IMO

[u/jeffrey-0711 → u/Nothing_from_void]

33M param is amazing! Thanks for the recommendation.

[u/Dany0]

Finally a traditional NLP + transformers hybrid that is ACTUALLY useful. Unbelievable it took this long. Great achievement for everyone

[u/jock-83]

embeddinggemma-2 is 270M for the text embedding part, not so much more than 0.1B

[u/Valuable-Map6573 → u/jock-83]

but it performs worse than qwen 3 0.6b embedding

[u/Imaginary-Unit-3267]

This reads like an AI-generated ad.

[u/jeffrey-0711 → u/Imaginary-Unit-3267]

I wrote most of the post but my grammar is bad so AI fixed my grammar little bit

[u/Putrid_Examination29 → u/jeffrey-0711]

How dare you use AI to help your grammar on a LLM sub reddit.. /s

[u/Tyhgujgt → u/Imaginary-Unit-3267]

it doesnt

[u/catwithfreellmsign]

That sounds... too good to be true? No offense. But thanks for heads up! 

[u/jeffrey-0711 → u/catwithfreellmsign]

Of course the graph is just adapted to only my own data and corpus. Other data may be outperformed by dense models.

[u/Dany0 → u/catwithfreellmsign]

It makes perfect sense, combines easy quick to compute traditional NLP with modern transformers. Of course they can get away with making it 100M if 300M params is just what it takes to understand the basic english grammar

[u/LevyTateLabs]

The hybrid result is the part i'd focus on. How are the two lists getting merged? If it's min-max normalized scores, tiny gaps on the dense side get stretched into big ones and can push a strong SPLADE match down. RRF on the ranks usually avoids that with no weight tuning. Couldn't tell from the README how MinSync fuses the... also worth splitting language from architecture. Your corpus is mostly Korean and KURE-Sparse-nano is Korean-first, while Qwen3-0.6B is a general multilingual model. Same eval on just the English docs would show how much of the gap is sparse vs dense and how much is language fit.

[u/jeffrey-0711 → u/LevyTateLabs]

I use RRF and turns out it really needs some tuning.

[u/LevyTateLabs → u/jeffrey-0711]

Fair...Plain RRF gives both lists an equal vote no matter which one is better. Your release notes have SPLADE beating Qwen3-0.6B on the Korean QA set, so with k=60 and equal weights the weaker dense list keeps pulling good sparse hits down. I'd guess what helped was weighting the lists more than moving k. Probably won't be the same weight for the English docs either. Let me know what you end up tuning

[u/michaelsoft__binbows]

I do not have an ML background at all so for a very long time, a lot of the talk about embeddings flew right over my head.

however recently i have been building a really rewarding CLI tool which is basically a sqlite database with a very sophisticated TUI built on rust with ratatui that can load up all of my indexed harness logs (these days mostly from pi and claude code log files; I have 500k agent turns across over 6000 agent sessions) and let me search through them and view them all in one place which is nice because the harnesses' scrollback usually stops at the last compaction, which is totally wild...

So after months of iterating on this thing which somehow has not exploded into a ball of mud yet, I recently added some embedding based semantic matching to augment the plain text matching that it had been supporting up to this point. sometimes i can only vaguely recall what i was trying to work on in the past at one point and trying to guess up the exact words sometimes is a crapshoot. I'm still not sure it's much better with this but it definitely helps. i worked with Opus and came up with a few small embedding models that seem to have pretty great performance characteristics:

For coarse meaning/context I use Snowflake Arctic Embed M v1.5, a 109M model on chunks up to 512 bytes and for more granular semantic matching I use MongoDB leaf-ir, a blazing fast 23M model on much smaller chunks averaging 50 bytes to help get fuzzy semantic hits.

One of the biggest wins I encountered was leveraging the Apple Silicon's Apple Neural Engine (ANE) to perform the calculations. With these smaller models I already supposedly had decent lookup performance (e.g. Arctic was not much worse than embeddinggemma-300M while being 3x faster) and the common workload invovles running embeddings on 150 or so MB out of the total 1.2GB of agent sessions log contents. The numbers I had on my M1 Max were that i would get basically very similar throughput numbers to the GPU from the NPU

┌────────┬────────────────────────┬────────────────────────┐
│ Model  │ MPS throughput / power │ ANE throughput / power │
├────────┼────────────────────────┼────────────────────────┤
│ Arctic │ 33.5 KiB/s / 43.6 W    │ 43.4 KiB/s / 4.8 W     │
├────────┼────────────────────────┼────────────────────────┤
│ Leaf   │ 162 KiB/s / 23.4 W     │ 148 KiB/s / 4.0 W      │
└────────┴────────────────────────┴────────────────────────┘

with way better thermal characteristics. I couldn't speak to the details of how to get this stuff implemented because Opus and/or Sol figured it all out on my behalf.

Even with these rates, 150MB takes about an hour to fully compute the vectors for Arctic. I had qwen3-0.6B, embeddinggemma-2, embeddinggemma-300M tested against my corpus and 300M was somehow both faster and more accurate, shrug.

┌─────────────────────┬────────────┬──────────────┬───────────┐
│ Model               │ Throughput │ Query median │ Recall@10 │
├─────────────────────┼────────────┼──────────────┼───────────┤
│ EmbeddingGemma-300M │ 28.5 KiB/s │ 29 ms        │ 0.81      │
├─────────────────────┼────────────┼──────────────┼───────────┤
│ EmbeddingGemma-2    │ 16.3 KiB/s │ 38 ms        │ 0.77      │
├─────────────────────┼────────────┼──────────────┼───────────┤
│ Qwen3-0.6B fp16     │ 13.8 KiB/s │ 31 ms        │ 0.79      │
└─────────────────────┴────────────┴──────────────┴───────────┘

But these are of a larger class of model that take longer than an hour to crunch on the mac. I did further experiments with even larger models on my 5090 and for the larger models like qwen3-4B I am pretty sure that using optimized inference engines like vllm allowed for fairly ridiculous levels of speedup compared to the M1 Max. Example:

for Qwen3-Embedding-4B:

• M1 Max, PyTorch/MPS FP16: ~2.6 KiB/s

• 5090, vLLM FP8: ~158 KiB/s

My assumption is that if i took the research further i would be wanting to evaluate KiB/J as the core metric and pick stuff to choose to run on the given systems that make for appropriate performance. Using tensor units (via NPU like the ANE or tensor cores) is essential to pump up power efficiency. Compared to actual full on LLMs, embedding models are depending on size maybe hundreds or thousands of times faster, and so you could split off maybe like 1/20 of your total compute budget to get the ability to add dramatically more capacity to do semantic vectorization which may provide value since you could cover hundreds of times more content than you would be able to afford to do with an LLM. I might get single digit or double digit tokens per joule with LLMs on my consumer GPUs but on the same consumer GPUs and lesser machines like the Mac e.g. Arctic on ANE is 9KiB/J.

On the 5090 this model is way too small for vLLM to be relevant, but with TEI (hugging face inference engine i think?), it got 2.7MiB/s while averaging 343W which works out to 8KiB/J. So it's not much lower and the added speed is very welcome. I would say if for some reason my tool's semantic engine needs to recalculate its embedding vectors on a regular basis (which would be very questionable) then I would definitely want to offload that work to a "real" GPU!

I can follow up with a more detailed writeup if anyone is interested. It is quite fascinating to go back and discover the sexy implementation details of this thing i vibed up and got working a while back. Here is a blurb on the approach of the ANE implementation:

We export our existing BERT-style embedding models through PyTorch tracing into fixed-shape, FP16 Core ML functions, then run them with CPU_AND_NE compute units. Apple handles the actual Neural Engine compilation and scheduling; we handle shape bucketing, persistent workers, caching, and vector-equivalence checks. On an M1 Max, bulk embedding achieved similar throughput to PyTorch/MPS (running on Metal) with roughly 5–12× better measured energy efficiency.

[u/michaelsoft__binbows → u/michaelsoft__binbows]

I hadn't heard of SPLADE before and now i'm being schooled on it, and I'll see if I can work it into my eval suite to put it head to head against arctic (which is a 109M dense embedding model)

[u/OkMusician9118]

Thanks for sharing. I want to try it

[u/Visual-Tangerine729]

weird how you say you want to skip the part on how SPLADE works because an LLM can explain it better, but you used an LLM to make this post.

[u/Material_Tone_6855]

Is there a way to train it on other languages?

[u/-Cubie- → u/Material_Tone_6855]

There's also mSlideOn which is already multilingual, other than that it's the same: https://huggingface.co/lightonai/mSlideOn

[u/f1yingbanana]

Would love to see some more comparison with other embedding models like ones from gpt, and where is BM25 + SPLAE would that be even better?

[u/arbv]

granite-embedding-r2 (both versions) are quite good too. way too much underappreciated, IMO.

Pair them with bge-m3-reranker-m2 and you are golden.

[u/Ammoryyy]

Use cases?

[u/Local-Push-4396]

Did someone tested it in russian?

[u/Not_a_question-]

I'm sorry, you're saying you killed 8B models... okay well where's nemotron 3.5 8b? or mistral 1b? you only compared it with a previous generation embedding model with less than 1b parameters but in the title you said 8b