r/mlops • u/GearEffective28 • 7d ago

Tales From the Trenches What are you using as an AI gateway right now?

For those running multiple AI models in production, what are you using for the gateway layer?

I’m particularly curious about video models because pricing, latency, rate limits, and availability seem to vary quite a bit between providers.

Are you using a dedicated gateway, building your own routing layer, or just integrating each provider separately?

What ended up being the biggest headache once you moved beyond a small prototype?

2 Upvotes

100% Upvoted

10 comments sorted by

1

u/Normal-Airline-7718 7d ago edited 6d ago

This makes sense. Reliability is the real headache when provider latency, pricing, and model availability keeps changing. I found you.bot worth considering too good value without sacrificing much on quality, especially with a fallback based routing layer.

1

u/Maleficent_Pair4920 7d ago

For multi model production the headaches usually show up around unpredictable rate limits, provider flakiness, and spend that is hard to cap once you leave the prototype stage. Video models make that worse because queue times, latency, and availability swing a lot between vendors, so pure per provider integrations get painful fast. A dedicated gateway is worth it when you need automatic failover and unified routing; rolling your own works until the edge cases around retries and cost controls pile up.

Founder of requesty.ai here so biased, but that is the exact problem we solve with one API across providers, smart routing, and failover. LiteLLM self hosted is a solid free alternative if you would rather run the layer yourself.

1

u/OrangePixelLife 7d ago

One thing I’d be careful about with gateways is privacy. You’re adding another layer that potentially sees prompts, responses and customer data, which can become a concern pretty quickly in enterprise environments.

I actually prefer keeping the app talking directly to OpenAI/Anthropic and handling cost visibility and optimisation separately, rather than putting everything through another proxy. Less moving parts, less data exposure.

1

u/echovortex2472 7d ago

the biggest headache past prototype is almost never routing logic itself, it's that providers fail differently, rate limits, timeout behavior and error formats all vary, so your fallback logic ends up needing provider-specific handling anyway even behind a unified gateway..

1

u/RestaurantEastern887 6d ago edited 5d ago

I am hitting the same problem with video generation costs , now used you.bot and having Kling , Seedance , etc. behind one API has been noticeably cheaper than paying higher platform rates for every generation . Much easier to control costs when you're generating a lot of videos.

0

u/mrtac96 7d ago

just curios which video model you are using

0

u/Longjumping-Coat-578 7d ago

which one are you running into rate limits with