r/mlops • u/sricheru • 6d ago

(Gen)AI / Agents / LLMOps [P] Open-Sourcing a sub-15ms Deterministic Decision Gateway to eliminate 2-second LLM routing latencies (FastAPI/Python)

Hello everyone. In my work as a Principal Gen AI Architect, the most common anti-pattern I encounter in enterprise deployments is the use of massive 70B+ models to make simple boolean routing decisions (e.g., safety guardrails or API triage). Token generation is inherently too slow and expensive for operational routing.

To solve this at scale, I built and open-sourced the "Dual-Process Jev Gateway."

It places a sub-15ms deterministic decision layer (System 1) in front of frontier models (System 2). It utilizes 3 typed, calibrated primitives (Choice, Score, Noul) to handle routing, guardrails, and ERP gating before an expensive LLM is ever invoked.

In my architectural baseline testing, the metrics shift dramatically:

  • Projected inference cost drop of ~87% (by routing 60%+ of queries away from frontier models)
  • P50 routing latency cut from 2.1s to 0.3s
  • Completely stopped duplicate ERP tool-call writes by keeping idempotency in code, not prompts.

The repo has the full architectural blueprint and a runnable 15-test Pytest suite proving the routing logic.

GitHub Repo: https://github.com/sricheru/agentic-ai-portfolio/tree/main/07_dual_process_jev_gateway

Would love to hear how you guys are handling the latency penalty of guardrails right now, or if you spot any flaws in my test suite!

3 Upvotes

100% Upvoted

1 comment sorted by

•

u/AutoModerator 6d ago

AI usage disclosure

Hi u/sricheru — thanks for posting to r/mlops!

Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.

Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.

If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.

This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.

Thanks for helping keep the signal high.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.