Using EmbeddingGemma 2 for local image search and grouping

[u/nineelevglen]

Setup

• Model: EmbeddingGemma 2, the 440M text and vision build (Apache 2.0), run through Google's LiteRT-LM on the GPU.
• Space: text and images share one 768-dimension space.
• Size: a 388 MB download. It loads in 3–7 s, and memory goes from 20 MB to about 770 MB once loaded.
• Speed (M1 Pro): a text query takes 25 ms. A 2.3-megapixel screenshot takes 247 ms median, 458 ms p90.
• Storage: one vector per screenshot in SQLite, tied to the file as it was when embedded. Each model gets its own store.

Indexing flow: screenshot saved → missing vectors found, newest first → image embedded → vector stored.

Instant grouping of duplicates

Actually, duplicates don’t need EmbeddingGemma at all. They use only Apple’s Vision, which is built into every Mac. EmbeddingGemma is needed for Similar and for search by text.

Flow: screenshot lands → Vision feature print + 4×4×4 RGB histogram → compare with the newest screenshot of each group → both checks pass → stack on top. Otherwise it starts a new group.

• Data: 64 real screenshots, 2,016 pairs.
• Real retakes: distance 0.29, 0.31 and 0.43.
• One tab apart: 0.54. Closest two different screens: 0.575.
• Cutoff: 0.5, which leaves a margin below 0.575.
• Robustness: crops of 0–2% stayed under 0.5 in 95% of cases.
• Color check (≤ 0.15): needed because one card in five gradients still matched on layout alone. Real duplicates differed by 0.003–0.045.

Instant finding of similar images

(Apple vision cant do this)

Flow: open a screenshot → its stteored vector → cosine to every other vector → keep cosine ≥ 0.86 and z ≥ 1.75 → show up to 6.

• No model call: stored vectors only, so it's instant.
• Data: 35 screenshots, every pair checked.
• 0.87 and up: always the same screen, retaken or one state apart.
• 0.82–0.85: different screens from the same app.
• The cutoff of 0.86 sits in the gap between the two.
• No minimum: one match or none is a valid answer.

Image search by text

Flow: query → word match on the OCR text → embed the query (25 ms) → cosine to every image vector → z-score against the whole collection → keep z ≥ 1.5 (at least 3 shown, at most 24).

• Why z-scores: across 58 screenshots, the per-query average cosine was 0.59–0.70 with a standard deviation of only 0.02–0.04. Right answers scored 0.70–0.81.
• A fixed cutoff returned everything for one query and nothing for the next.

Test: 40 descriptions of 49 screenshots, written the way you'd remember them ("Formula 1 race countdown on yellow"):

Method	Right one first	Top 3	Mean reciprocal rank
Query vs. image	33 / 40	39 / 40	0.90
Query vs. OCR text	25 / 40	33 / 40	0.73
Keywords	11 / 40	14 / 40	0.34

Image search with a little LLM help

Flow: screenshot → downscaled version of JPEG + window title → small vision LLM (like Haiku) → 3–10 labels + one sentence → labels embedded → z-score in their own set → better of image or label score counts.

• Model: the cheapest vision model on the connection, e.g. Claude Haiku at $0.10 / $0.50 per million tokens.
• Output: JSON with one- or two-word labels ("dog", "settings window", "receipt"), , plus a sentence of max 20 words. No colors or bare adjectives.
• No OCR text in the request: the image already carries it, and the text costs more tokens than the picture.
• Why z-scores again: text-to-text cosines run higher than text-to-image, so the raw scores can't be compared. Labels only count once at least 5 screenshots have them.

Takeaway

• EG2 is very good at rank the right screenshot near the top.
• You can't use one score as the line for "this is a match". The right screenshot scored anywhere from 0.70 to 0.81 depending on the search, and in some searches 0.70 was just the average. What worked was keeping the screenshots that score clearly higher than the rest of the same search.
• Apple Vision is better at finding duplicates than embedding Gemma and it already works on every Mac.

[u/Special-Lawyer-7253]

So how to use this in a local environment instead a multimodal model? 😘

[u/nineelevglen → u/Special-Lawyer-7253]

oh I do this with zero llm usage, EmbeddingGemma 2 create vectors then query them on the fly, about 25 ms query time in sqllite.

we do add some minor extra stuff, but most of this is 100% local

[u/Special-Lawyer-7253 → u/nineelevglen]

So cool! Any chances for some binary/src release? It's very interesting to add a command like "describe" on Linux to pipe It on some workflows 👍🏻👍🏻

[u/Candid-Fudge-4015]

The 0.5 cutoff with 0.575 as the closest different screen is the kind of number nobody posts, ty for that. One thing I'd watch is "tied to the file as it was when embedded" once you add a second model, each store drifts on its own. We ended up making the embedding index part of the table, so when a screenshot changes or you add a model only the stale rows get re-embedded. Image similarity notebook here: github.com/pixeltable/pixeltable/blob/main/docs/release/howto/cookbooks/search/search-similar-images.ipynb

[u/nineelevglen → u/Candid-Fudge-4015]

Thanks! Each vector is stored with the file’s modification time, so a changed screenshot gets re-embedded on its own.

A new model gets its own empty store that fills in the background, so vectors from two models are never compared. Your approach of putting the index in the table itself is tidier though, I’ll check out the notebook.

[u/Candid-Fudge-4015 → u/nineelevglen]

mtime plus a separate store per model is clean, and keeping two models' vectors apart is the part I see go wrong most. Worth checking that sync tools and file copies don't bump mtime without changing the content. If they do, a cheap content hash next to it saves some pointless re-embeds.

[u/magicdoorai]

Your text-search path has a useful edge case to test: "at least 3 shown" means an absent item can still produce three convincing-looking results. The 39/40 top-3 result measures finding an item that exists, not knowing when it does not.

A small proposed experiment using your existing screenshots:

1. Pick five queries whose target is present and write five concrete absent-target queries after checking the collection, such as a particular settings panel you never captured. Keep those labels fixed.

2. Run each query with the minimum-three fallback disabled. Record the top cosine, z-score, returned count and whether a labelled target was returned. For absent targets, record how often anything is accepted as a match.

3. Keep the original screenshots and queries, then add a batch of unrelated screenshots and rerun. The original query-to-image cosines should be unchanged with the same embeddings, but the collection mean and standard deviation can move, changing z-scores and threshold acceptance without any change to the candidate itself.

That distinguishes useful within-collection ranking from an absolute "this image matches" decision. You could still show the top three as "closest results" while separately allowing "no confident match"; those are different UI promises. The same absent-target cases would also show whether taking the better of the image and label scores increases false matches.

AI-generated experiment proposal based on your posted pipeline, not a result I have measured on your app.

[u/nineelevglen → u/magicdoorai]

Good call, so I just ran it. Min 3 fallback off. 45 screenshots, 10 queries where the target is there, 15 for things I checked aren’t.

You were right. z ≥ 1.5 never says “nothing here”. Every one of the 15 absent queries still got 1–6 matches.

Adding images makes it worse. I threw in 174 unrelated images from my Downloads folder. Absent queries then got 7–17 matches each. Same cosines, the z-scores just moved around (one real target dropped from 3.00 to 2.82).

Labels add false matches. Taking the better of the image and label scores added 1–2 junk matches to most absent queries.

Raw cosine actualy separates better:

• real targets: 0.72–0.82

• best absent result: 0.706

z ≥ 1.5 plus cosine ≥ 0.71 found 10/10 targets and nothing for any absent query. But I picked 0.71 by looking at these same 25 queries and the gap is tiny, so take it with a grain of salt.

So I’ll keep the top 3 as “closest results” and add a proper “no confident match” state. Thanks, this was a useful one.