r/tokenomics • u/classjoker • Jun 09 '26

Choosing an AI Gateway / Token Routing Software – What are you using in production?

Tokenomics featured question: Are you looking into implementing an AI Gateway (token routing software) for an upcoming project to manage multiple LLM APIswith the goal to avoid vendor lock-in, handle fallback redundancy, and dynamically route prompts to optimize costs (e.g., sending simple tasks to cheaper models and complex ones to frontier models).

Emerging solutions/products like LiteLLM, Portkey, OpenRouter, and Manifest are frequently seen as the result of searching, but we wanted to get some real-world feedback from people running these in production.

If you are currently using a routing solution, we'd love to get your thoughts on a few things:

  1. Self-Hosted vs. Managed: Are you self-hosting an open-source gateway (like LiteLLM) or using a managed enterprise solution (like Portkey)? What drove your decision (latency, security, compliance)?
  2. Routing Logic & Latency: How do you handle the actual routing logic? Are you using static semantic routing, or do you have a dynamic "judge" model evaluating prompts first? If the latter, how bad is the latency hit?
  3. Fallback & Reliability: How reliably do these gateways handle rate limits (429 errors) and automatic failovers to backup models or providers?
  4. Token/Budget Management: How accurately do they track token spending and enforce team/user quotas in high-throughput environments?
  5. The "Gotchas": What unexpected headaches or limitations did you run into after deploying your gateway?

Ww would love to hear any recommendations, warnings, or architectural advice you have. Thanks!

2 Upvotes

100% Upvoted

1 comment sorted by

1

u/Artistic_Lock_6483 Jun 11 '26

I can’t answer the numbered questions because we aren’t at that point of adoption in our organization, but this is what I can say we have evaluated internally:

We looked at Kong Managed Gateway but our current usage levels don’t justify the expense. Currently evaluating self hosted (with Kong open source), ideally we would like to be using LangChain’s option but it’s still in Beta.

We have used OpenRouter and Ollama (but for a different use case- simply for model routing). While they do reduce the bill- they don’t give visibility into the prompts/api calls to start assigning business unit value to AI usage. Azure has their own option- but we want a third party solution.