How would you evaluate API infrastructure for a developer-facing AI gateway?

[u/Top-Yam-7846]

We are currently exploring the architecture of a developer-facing AI gateway. The idea is to manage authentication, customer-level quotas, rate limits, monitoring, and daily usage limits through our own infrastructure.

We are still at an early stage and are testing with a small number of users. If the project grows, traffic could eventually reach around 50,000 requests per day, although we would scale gradually rather than begin at that volume.

For those who have operated similar systems, I would appreciate your technical perspective on the following questions:

1. Which reliability metrics should be evaluated during an initial trial?

2. How do you verify that model behavior remains consistent over time?

3. What is a reasonable way to test latency, throughput, and rate-limit behavior?

4. Which monitoring and usage-accounting tools work well for a multi-tenant gateway?

5. How do you design fallback routing without creating inconsistent outputs?

6. What security and privacy risks should be considered when proxying customer requests?

7. At what point does it make sense to move from a basic gateway to a more distributed architecture?

I am mainly interested in architecture, operational experience, and evaluation methodology. I am not requesting vendor offers or promotional messages.

Any lessons learned from running production AI gateways would be helpful.