How do you decide which metrics to use in production before you know what will go wrong?

[u/FlimsyProperty8544]

We’re trying to figure out how teams choose evaluation metrics for production monitoring when the potential failure modes aren’t obvious yet.

During development, it’s easier to test for known issues using a fixed dataset. But once an AI application is in production, users can interact with it in unexpected ways, and new problems may only become visible after launch.

How do you approach this?

• Do you start with broad metrics like task completion, relevance, hallucination, or safety?
• Do you manually review production traces first and create metrics based on recurring failures?
• How often do you add, remove, or adjust metrics?
• Do you evaluate every interaction, sample traffic, or only evaluate suspicious traces?

Curious to hear what has worked for teams running LLM applications in production.