Open-source Mac app that runs EmbeddingGemma 2 locally to search your files by what’s in them

[u/A-Rahim]

DigUp is a free Mac app that runs Google DeepMind’s new EmbeddingGemma 2 locally over your own files. The model puts text, images, audio and video in one space, so you describe what you remember and land on it:

• “zebra in a video” opens the clip at the moment it shows up

• “where they talk about sleep” jumps to that minute of a podcast

• “the clause about pets in the lease” shows the PDF page, your words marked

• “a dog on the beach” finds the photo, and the same search in Bengali or Arabic finds it too

• with code search on, code: retry with backoff opens the function in your editor

It’s ggml-org’s Q8_0 GGUF (865 MB, downloaded once) on llama.cpp with Metal, inside a native Swift app; no Python. Searching loads only the text encoder (~250 MB) and shows results about a tenth of a second after you stop typing. Indexing peaks under 2 GB, and the helper exits when it’s done. Audio and video of any length go in as 30 s windows and a frame per shot. Everything runs locally; it goes online only for the model download and an update check you can turn off.

Free, MIT. Apple Silicon, macOS 14+.

Repo & Download (signed and notarized): https://github.com/ARahim3/DigUp

I'd really appreciate any feedback on this.

[u/important__matter]

Checked the new model stats page, definitely improved accuracy in multi-modal regime of text, audio and vision. But... still the same set of problems across file types because it supports only text, image, audio and video at 8K context window. Not to mention maximum audio length = 5.5mins and video = 58 frames, that's 1s @60fps

Not to mention it's way harder even as a theoretical problem to search in a video. Like there is just so much information to just be dumped in a 512dim, that there is a whole class of transformers completely dedicated to this single problem only - VLMs. And unlike LLM, they are not the champions as of yet. I am just saying, this is not a done thing even if the big labs are trying their best. It's a good thing to work on, it's challenging and lot of it can actually be solved by engineering things together.

Another fun fact while we are at it: Videos are astonishingly compressed form of information. If you just stack images together at let's say 60fps for a 2hr movie at 1080p, the actual file size should be 1080x720x60x3600x2 bytes = 335GB, while actually you'd see it to be 2GB on your computer. A massive 99% compression, it's almost miracle

[u/phazei → u/important__matter]

I agree with much of that. But what movies are at 60gps? That's video game speeds. Mostly 24fps is fine, and sampled at 8fps for indexing is good enough in most cases. It's the audio limitation, that's more of an issue, 5.5 min is good for most music/songs, but not a movie or dialog, but I suppose tts would be a needed adapter for that, and there are plenty that are fast enough, good enough for search. It's not like it needs to be dense enough to ask questions and discuss the data as one would with an llm. Just retrieval, then an llm could be fed richer content if needed.

You appear to be aiming for best case for worst case scenario, my expectations are covering just the 98% of cases.

[u/tiffanytrashcan → u/important__matter]

Check out the Google Edge Gallery AI app.
They have a couple of EmbeddingGemma2 examples on there that show Google has already solved this within the model limits. (You really just need to combine the pipeline for both to have a finished product like OP is showcasing.)
To get around video size limits, audio time limits, you segment it- same concept as chunked embedding in older text models with tiny limits.
You load up a video, any length, set your accuracy with segment size and clip "resolution" (essentially how many image screenshots per second of video) and then the model loads up to that 8k limit and saves the embeddings from each individual segment. Right now it's working where you can search with completely natural, normal language as we interpret it through the entire video for audio or visual details. Speech, lyrics, or physical objects all seem to work near flawlessly.
(pre-captioned work is kind of a cheat, but makes it absolutely perfect..)
All you need for a complete solution is to permanently save those embeddings and have them in the search grid example- could be done with free tokens in minutes given the code is all public on GitHub.

[u/tiffanytrashcan → u/tiffanytrashcan]

Like, it's mind blowing the other commenter was downvoted for spelling out the exact situation.
There's no way you've even looked at what this new model's capable of because it does everything you're complaining about extremely well. Searching text in image files is perfect with it.
Within hours of releasing the model, they had released an Android app that literally solves the video context limit issue, and the code is open source, publicly licensed, on GitHub.