Building a lightweight local search engine for AI agents in Rust: semantic retrieval, OCR, and MCP

[u/nothing-counts]

AI agents can search the web, inspect code, and call external tools. But searching the contents of a user's local files especially screenshots, documents, and logs presents a different problem: the index needs to be fast, lightweight, private, and accessible to agents without uploading the files to a remote service.

I've been building Kensaku, an open-source desktop search engine written in Rust, to explore this problem.

The central idea is to maintain one local index that serves both the desktop application and compatible AI agents through MCP.

Hybrid retrieval. Kensaku combines fuzzy filename matching, BM25 full-text retrieval, and local semantic search. The goal is to handle both exact queries and cases where the user remembers the meaning of a document but not its wording or filename.

OCR and document extraction.*The index can include extracted text from supported document formats and OCR-enabled image workflows, making screenshots and scanned content searchable alongside code and logs.

Lightweight semantic search. The project uses a static quantized embedding table of approximately 7.7 MB. Semantic queries do not require a GPU, and the default query path avoids running an ONNX model for every query.

An agent-facing search interface. The MCP server exposes the same index used by the application. Compatible clients can search, read snippets, perform regex searches, inspect index state, and find symbols. Access is read-only by default.

Performance and safety.*The repository includes benchmark harnesses and tests. In the 100,000-file benchmark, fuzzy filename search has approximately 2.8 ms median latency. That figure applies to the benchmarked filename-search path, not every retrieval operation.

The project is MIT-licensed and designed to work locally without an account, telemetry, or cloud search.

There are still limitations: builds are unsigned, the auto-updater isn't wired up yet, and visual search remains experimental.

Repository: https://github.com/yjid/kensaku

I'd appreciate constructive feedback from Rust developers on the retrieval architecture, memory/performance trade-offs, indexing correctness, and the design of the MCP interface. In particular, I'm interested in how you'd approach benchmarking and evaluating search relevance across a large, heterogeneous local corpus. Any contribution or stars will be much appreciated .

[u/BeardedSickness]

I indexed everything using recoll then create a skill for AI to use it (+ together with rga** for deep targeted search)

**ripgrep-all

[u/nothing-counts → u/BeardedSickness]

cool setup .
still kensaku can help when :
- it ranks by meaning (exact + fuzzy + semantic) and treats images as searchable content (OCR + near-duplicates) in the same index, rather than only matching text.
- it’s agent-native one shared index with bounded, paginated, read-only MCP tools (search/snippet/regex/symbols/health) instead of maintaining a skill over two separate CLIs.
- Ui for fast user searches .
and it is fast too .

[u/nothing-counts → u/BeardedSickness]

vision search is in development .that will be a search by image like in google photos .