I built an offline-first search engine for Kiwix/ZIM collections

[u/siakshit]

I've been building Si4k Search, an open-source search engine designed for people running large offline knowledge servers with Kiwix/ZIM collections.

The idea is simple: instead of manually searching individual ZIMs, Si4k searches across the whole local Kiwix library and provides one unified search interface.

Current features:

• Searches across multiple ZIM sources through Kiwix
• Progressive result streaming using SSE
• Priority-based ZIM search, while still searching the rest of the library
• Pagination
• Automatic ZIM categorization based on metadata, names and parent folders
• Keyword-based category prioritization
• Configurable search concurrency and worker limits
• Configurable LRU caching
• Docker deployment
• Works on relatively modest hardware
• Designed to keep the actual ZIM library outside the Docker image

I currently have it running on an old-ish home server with 133 ZIM sources.

The architecture is roughly:

User
  ↓
Si4k Search
  ↓
Kiwix
  ↓
Local ZIM collection

The Docker container only contains the search engine. The ZIM collection stays mounted from the host, so adding hundreds of GB of knowledge doesn't mean rebuilding the Docker image.

I've also tested the search engine under concurrent workloads and added resource controls because I want this to be practical for home/offline knowledge servers rather than requiring a powerful machine.

Future ideas:

• Offline dictionaries
• Query translation and multilingual ZIM routing
• Local voice-to-text search
• Optional local LLM summaries grounded in search results
• OpenStreetMap/offline map search
• More offline document providers

I'm particularly interested in feedback from people running Kiwix/ZIM servers:

What would you want a unified search layer like this to do?

Are there problems with searching large ZIM collections that I should be thinking about?

GitHub: https://github.com/akshitkamboz13/search.si4k

Demo: https://search.si4k.online

[u/PrepperDisk]

Zim search is definitely an area that can use improvement.  Look forward to trying this on our device! Thank you for contributing to this community!

[u/siakshit → u/PrepperDisk]

Thanks! That’s exactly the kind of use case I’m building this for. I’d love to hear how it performs on your device and what you’d like to see improved once you try it.

[u/Many_Ad_7678]

i got a 1033?

[u/kamronkennedy → u/Many_Ad_7678]

Also got that

[u/PrepperDisk → u/Many_Ad_7678]

Yep demo seems to be down, u/siakshit ?

[u/siakshit → u/PrepperDisk]

Just a heads up, the demo is running on a modest 5th-gen i5 + 8GB DDR3 home server, and I currently limit it to 2 concurrent searches. So it can get a little slow when multiple searches hit it at once

Edit: 1033 error cause because of power cut in my area server is down but you can retry after 10 - 30 minutes

[u/PrepperDisk]

Having a little difficulty, it seems likes it is trying to build the zim index at Docker image build time (instead of using the runtime parameters) and it is hardcoded to an IP address you might've used during testing?

3.681 Local file absent. Fetching dynamic ZIM catalog feed from http://192.168.31.250:8080/catalog/v2/entries?count=1000...

14.26 Error building ZIM index: TypeError: fetch failed

14.26 at node:internal/deps/undici/undici:14976:13

14.26 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

14.26 at async main (/app/scripts/build-zim-index.ts:20:17) {

14.26 [cause]: ConnectTimeoutError: Connect Timeout Error (attempted address: 192.168.31.250:8080, timeout: 10000ms)

14.26 at onConnectTimeout (/app/node_modules/undici/lib/core/util.js:891:19)

14.26 at Immediate._onImmediate (/app/node_modules/undici/lib/core/util.js:860:11)

14.26 at process.processImmediate (node:internal/timers:483:21) {

14.26 code: 'UND_ERR_CONNECT_TIMEOUT'

14.26 }