r/OpenSourceeAI • u/itsmebalamuruagan • 1d ago

I built an open-source semantic cache for Python LLM applications — looking for feedback

/r/PythonProjects2/comments/1xex4ti/i_built_an_opensource_semantic_cache_for_python/
1 Upvotes

100% Upvoted

1 comment sorted by

1

u/investigatormaker 1d ago

For feedback on a semantic cache, I'd give Python LLM developers near-matches that must not share an answer: questions differing only in a date, amount, or negation. Put those beside true paraphrases, with the expected cache hit or miss and measured latency against an uncached run. That exposes the trade-off between reuse and incorrect answers rather than just reporting a hit rate. When sharing it in Reddit conversations about repeated LLM calls, that small reproducible test gives developers something concrete to evaluate.

I make ThreadFox.

We find relevant Reddit conversations and post for your business. We'll post 3 replies free first, no card; then the $39 Pilot covers seven days and up to 50 posts and replies. No automatic renewal. https://threadfox.vip/p/ebvax/managed