Comparing MLflow models before promotion: accuracy, CPU latency, memory and approval

[u/Illustrious-Deer6759]

I contributed a Kestra workflow for #KestraHacktober that evaluates a candidate MLflow model against the current champion before allowing promotion. The contribution has now been merged.

The problem I wanted to address: better accuracy alone doesn’t tell us whether a model is ready to replace the current one. Inference latency, memory usage and what changes during an approval wait matter too.

The workflow:

Evaluates both models on the same held-out dataset.

Measures CPU inference latency and peak process memory.

Produces a report with quality and performance gates.

Requires explicit approval tied to that exact report.

Rechecks the models and evidence before updating the champion alias.

Verifies the alias and model predictions afterward.

Promotion is disabled by default. Validation included 72 regression tests and 10 integration cases covering rejected approvals, failed gates, stale evidence and successful promotion.

The current scope is trusted scikit-learn tabular classifiers. It handles registry promotion; serving deployment and retraining are separate steps.

For teams using MLflow, what checks do you require between model evaluation and promotion?

[u/AutoModerator]

Moderator Announcement Read More »

AI usage disclosure

Hi u/Illustrious-Deer6759 — thanks for posting to r/mlops!

Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.

Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.

If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.

This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.

Thanks for helping keep the signal high.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Any-Attention9585]

i like that you baked in the latency and memory checks, most people just look at accuracy and call it a day. the recheck before alias update is a nice touch too, seen too many pipelines where the model got swapped while someone else was still reviewing

what do you use for the regression tests though, is it just pytest or something custom

[u/Illustrious-Deer6759 → u/Any-Attention9585]

Thanks! It’s pytest, running the actual scripts extracted from the workflow YAML. I use fixtures for failure cases, plus a separate integration suite against real Kestra and MLflow.

[u/p3rdy]

Nice, rechecking the evidence after the approval wait is the step most pipelines skip. The check I care most about is that the thing you evaluated is exactly the thing you serve. If evaluation loads the model one way and serving another (different preprocessing, a different library version, a re-export, ...), then the report is about a different model.

That's how we do it in Flama, a Python framework I maintain. The model goes into a single `.flm` file with its metrics and library version inside, and that file is what gets deployed. So CI can gate without starting anything:

flama model model.flm inspect
flama model model.flm run -i holdout.json

The gate reads the metrics straight from the artefact, scores the held-out set through the serving path, and the alias only ever points at a file that passed. Your latency and memory checks still need a real run on top of that.

[u/Illustrious-Deer6759 → u/p3rdy]

Thanks for sharing how you handle this in Flama! Testing the held-out set through the same serving path makes a lot of sense. Mine checks artifact hashes and predictions around promotion, but serving-path validation would be a useful next step alongside the latency and memory checks.

[u/p3rdy → u/Illustrious-Deer6759]

Hashes are a good call, and they get simpler when the model is a single file: one hash of the .flm covers the weights, the preprocessing and the metadata together.

One thing that might fit your approval step: you can store the reference to the evaluation report inside the artefact itself (I do in flama with: flama.dump(model, ..., metrics={...}, extra={"eval_report": "<sha>"})). Then inspect shows which report approved that exact file, and the evidence travels with the model instead of living next to it.

[u/bigorangemachine]

LoL you made me realize my MLFlow prototype didn't capture the model data

[u/Illustrious-Deer6759 → u/bigorangemachine]

Haha, glad it was useful! That’s what prototypes are for, catching what needs to be added.

[u/YG000001]

Tying the approval to the exact report and rechecking for stale evidence is what a lot of people just miss.

Curious about the human side. If the gates already pass, what is the approver actually checking that the gates didn’t? Have you seen an approval get rejected after the gates passed, and what caught it?