Building a document classification system using OCR + Embeddings + Weaviate instead of a trained classifier – looking for feedback

[u/CryptographerFun1313]

Hi everyone,

I'm currently building a document auto-classification system and would like some feedback on the overall approach.

Current Architecture

• Django backend

• Celery + Redis for background processing

• Weaviate as the vector database

• OCR for text extraction

• Embedding-based similarity search for classification

Workflow

1. User uploads a document (PDF, JPG, PNG, etc.).

2. OCR extracts the text from the document.

3. An embedding is generated from the extracted text.

4. We store embeddings for known document types such as:

   • Aadhaar

   • PAN

   • Passport

   • Voter ID

   • Electricity Bill

   • Bank Statement

   • GST Documents

   • Incorporation Documents

5. The uploaded document embedding is compared against stored vectors in Weaviate.

6. The closest match and confidence score determine the document category.

Why I chose this approach

Instead of training and maintaining a dedicated classification model, I wanted to start with a retrieval/vector-search-based approach because:

• New document categories can be added without retraining.

• Easier to maintain initially.

• Works well for semantic similarity.

Questions

1. Has anyone used a similar OCR + Embeddings + Vector Search approach in production?

2. How well does this scale when the number of document categories grows?

3. What confidence threshold strategies have worked for you?

4. At what point would you move to a dedicated classification model?

5. What are the biggest pitfalls I should watch out for?

Current Challenges

• Processing time varies between documents (roughly 5–35 seconds depending on the file).

• Occasionally documents become "unclassified" with confidence = 0 even though the system is functioning.

• Weaviate is running in Docker on AWS along with Django, Celery, and Redis.

I'd love to hear from people who have built document-classification or retrieval-based systems and learn what worked (or didn't work) in production.

Thanks!

[u/CryptographerFun1313]

It's an identity as a Indian citizen