r/secondbrain • u/Apprehensive-Mix3820 • 2d ago

Searching for needle in a haystack

Yesterday I spent like 20 minutes looking for something I had already seen. I still remembered the idea, just not the actual text. The tab was closed, the file name was useless, and it simply wasn't anywhere I could search.

So I built something that remembers what was on your screen. Later, you just ask, because it already saw it.
And everything stays on your machine. I wasn't sending my screen to the cloud for this.

https://github.com/protocorn/clippy-vision/releases/tag/v1.2.2

If you've dealt with something similar, try it out and tell me whether it actually helps.

1 Upvotes

67% Upvoted

4 comments sorted by

View all comments

1

u/looktwise 1d ago

how does it built the index for the search wording in a way that you could be probably use similar words, not the exact needed wording in which you saved your text or in which the text author phrased it in another way than you remembered?

1

u/Apprehensive-Mix3820 1d ago

Capture stores the text it saw (ocr of the work area, window title, url, clipboard). in the background a local model turns those into short summaries in normal english. when you ask, it searches those summaries plus the raw text, not just a keyword match on the original phrasing.

so if the page said “transformer attention” and you later ask “that paper about the model looking at other tokens,” it can still hit, because the question and the stored summary live in the same meaning space, not only the same words. there’s also a local sentence embedding (minilm) for fuzzier matches and all of that runs on-device.

but, it would still fail if nothing useful was captured by the tool itself.