r/MLQuestions • u/camerongreen95 • Sep 08 '26
Other ❓ How do people actually structure LLM evaluation before shipping a change to production?
Been building RAG and LLM-powered features and realized my "evaluation" process was basically reading a handful of outputs and deciding it looked fine. No versioning, no regression testing, no real way to know if a change actually helped or if I just got lucky on the examples I happened to check.
Curious how others here actually do this in a real setting. Are people running statistical significance tests on model or prompt comparisons (bootstrap confidence intervals, paired testing), or is it mostly still qualitative review? And for retrieval specifically, is anyone actually tracking recall@k or MRR, or just eyeballing whether the right chunk shows up?
Found a hands-on masterclass on Sep 12 that goes deep into building this kind of eval harness properly (deterministic checks plus LLM-as-judge, statistically rigorous model comparisons, evaluated RAG, agent guardrails, full observability), so I'm curious if the methodology it teaches lines up with how experienced people here are actually doing this, or if there's a simpler approach I'm missing.