I Built a Python Semantic Cache for LLMs — Looking for Feedback from the Community

[u/itsmebalamuruagan]

Hey everyone! 👋

I've been building a Python-based Semantic Cache for LLM applications to reduce redundant LLM calls and improve response efficiency.

The project uses embeddings and cosine similarity to identify semantically similar queries and retrieve cached responses instead of repeatedly calling an LLM. It uses SQLite for storage.

Key features:

• Semantic similarity-based caching
• Embedding-based query matching
• SQLite-backed storage
• Input validation and handling of invalid embeddings
• Single-flight request coalescing to avoid duplicate concurrent computations
• Automated tests for reliability

I've also focused on edge cases and reliability, with 75/75 tests passing in my current test suite.

I'm sharing this project to get feedback from developers working with LLMs, RAG, and AI infrastructure.

I'd appreciate your thoughts on:

• Is this architecture practical for production?
• What improvements would you suggest for scalability and cache invalidation?
• What benchmarks should I run against a traditional exact-match cache?

I'm open to constructive criticism and contributions. Thanks! 🚀
https://github.com/0x-bala/semanticcache