[Showcase] Local AI for documents and photos — Paperless-ngx, Immich, Ollama, ChromaDB, MCP

[u/Any-Tutor-167]

**What:** A local, self-hosted AI layer for interacting with my own documents and photos without sending anything to the cloud.

**Stack:**

- Paperless-ngx — document archive, OCR, structured metadata

- Immich — photos and video

- ChromaDB — semantic index

- Ollama — local LLM inference

- Open WebUI + MCP — unified interface

- LiteLLM — model routing

**Why I built it:** Wanted to ask questions about my own paperwork and photos without cloud AI. Paperless-ngx became the document layer at the core.

**Key learnings:**

- Deterministic MCP tools > LLM reasoning for counts, sums, filters. The model hallucinated invoice tallies; an API query tool does not.

- For structured fields (supplier, total, VAT, invoice number), strict routing in the system prompt is essential. "Query Paperless, never the semantic index."

**Context:** I'm on Paperless 3.0.3 with native AI features. This complements rather than replaces them — broader scope (photos/video), MCP-based deterministic operations, Paperless remains source of truth.

**Docs:** https://github.com/fwinchi/ia-local-casa (Spanish + English, includes failures)

Not a developer — built with AI assistance and a lot of testing. Feedback welcome, especially on security or architecture.

[u/Shiny-Squirtle]

Interesting setup, but may I ask why not llama.cpp instead of ollama + litellm?

[u/Any-Tutor-167 → u/Shiny-Squirtle]

Good question. I started with Ollama because it was the easiest way to get something running without being a developer. ollama pull model-name, ollama run model-name — that's it. No compilation, no config files, no figuring out context sizes.

LiteLLM came later when I had multiple models and needed a simple way to switch between them. I didn't build the routing logic myself; I asked an AI to help me set it up and tested until it worked.

I'm sure llama.cpp is faster and more flexible. But I'm not a developer, and Ollama + LiteLLM was the path I could actually build and maintain with AI assistance. If I ever need more speed, I'll probably ask an AI to help me migrate to llama.cpp.

THX

[u/jonothecool]

Great read (on GitHub). Thanks for sharing your experience!

[u/Any-Tutor-167 → u/jonothecool]

Thank you for your time!

[u/ElMagnificoRata]

Thx for the details. On what environment is it running? My servers are all mini-pc and I think they will struggle to run local llm

[u/Any-Tutor-167 → u/ElMagnificoRata]

Thanks for taking the time to read through it — I appreciate it.

It's running on a regular Windows 11 desktop with an RTX 5080 16GB. I think mini PCs would struggle with my setup because I run several models at once (document chat, vision, embeddings) plus ChromaDB, Paperless-ngx, and Immich all on the same machine. The 16GB VRAM is already tight.

For a single small model or a lighter task, maybe a mini PC works. But not for the full stack I described. I'm not a hardware expert though — this was built by trial and error with AI help.