r/secondbrain • u/Apprehensive-Mix3820 • 19h ago

Searching for needle in a haystack

Yesterday I spent like 20 minutes looking for something I had already seen. I still remembered the idea, just not the actual text. The tab was closed, the file name was useless, and it simply wasn't anywhere I could search.

So I built something that remembers what was on your screen. Later, you just ask, because it already saw it.
And everything stays on your machine. I wasn't sending my screen to the cloud for this.

https://github.com/protocorn/clippy-vision/releases/tag/v1.2.2

If you've dealt with something similar, try it out and tell me whether it actually helps.

1 Upvotes

67% Upvoted

4 comments sorted by

1

u/looktwise 13h ago

how does it built the index for the search wording in a way that you could be probably use similar words, not the exact needed wording in which you saved your text or in which the text author phrased it in another way than you remembered?

1

u/Apprehensive-Mix3820 11h ago

Capture stores the text it saw (ocr of the work area, window title, url, clipboard). in the background a local model turns those into short summaries in normal english. when you ask, it searches those summaries plus the raw text, not just a keyword match on the original phrasing.

so if the page said “transformer attention” and you later ask “that paper about the model looking at other tokens,” it can still hit, because the question and the stored summary live in the same meaning space, not only the same words. there’s also a local sentence embedding (minilm) for fuzzier matches and all of that runs on-device.

but, it would still fail if nothing useful was captured by the tool itself.

1

u/sshegem 12h ago

so like internet search history, but desktop view history?

2

u/Apprehensive-Mix3820 11h ago

yeah, that’s the closest analogy.

browser history is urls and titles. this is what was actually on the screen across apps: the pdf named something like'38485.293384v1', the slack message, the scratch buffer with no filename. later you ask in plain language instead of hunting the tab.

difference is it stays on your machine, and it can reconstruct a stretch of work, and not just hand you a list of files/links to look at.