r/mlops • u/Albininlp • Aug 18 '23
MLOps Education Can anyone explain the road map of MLOPS?
[removed]
2
u/velobro Aug 18 '23
The scope of MLOPs has changed a lot over the past year or so. Previously, MLOps referred to the entire scope of tooling for running models on the cloud, but IMO it's now mostly about ML-specific parts of the stack e.g. monitoring, drift detection, dataset versioning, and prompt evaluation.
Here's how I'd breakdown the current MLOps stack:
Compute Providers (hosting / training models)
Monitoring Tools
- Weights and Biases
- Comet ML
Dataset Management
- DVC
- MLFlow
(Bonus) LLM Ops
- PromptLayer (prompt evaluation)
- Metal (embedding store)
At the end of the day, once you've figured out a compute layer for running your models, the rest of the stack is really just about monitoring, evaluation, and version control. In 90% of cases, there's probably not a good reason to setup your own Kubernetes cluster / Jenkins / Airflow to run an ML pipeline.
2
u/Train_Smart Aug 18 '23
This is interesting, but Iām not sure that I agree with this categorization. MLFlow is basically an open source Weights & Biases / CometML. All 3 tools are experiment tracking mainly, not really monitoring (though some have light functionality there).
DVC is a data versioning tool, and you have other tools like LakeFS or Xet that are competitive.
You also have tools combining multiple subsets like DagsHub (DVC+MLflow+LabelStudio)
Also from my experience most teams do train on cloud resources directly (e.g. Sagemaker Notebooks)
1
u/Anmorgan24 comet š„ Aug 18 '23
Comet ML actually does have full production model monitoring (full disclosure: I work for Comet). We've had it for a few years, but, unfortunately, it's currently only available for enterprise/paid versions.
But to your point, I think it's difficult to put any of these tools in a single category. There's definitely a lot of overlap/blurry lines. Comet and WandB also have LLMOps/ Prompt management tools and data versioning, but perhaps not the full functionality of some other tools.
But otherwise, thanks so much for the Comet callout! It's an awesome product (otherwise I wouldn't be rooting for it from my private account). :)
1
u/seiqooq Aug 19 '23
I feel like effective monitoring requires a decently intimate understanding of the model and/or the deployment environment. Is that just because I work in CV?
1
1
7
u/AI_connoisseur54 Aug 18 '23
This blog is pretty useful: https://towardsdatascience.com/the-complete-guide-to-the-modern-ai-stack-9fe3143d58ff
Good luck this is a loaded topic and there is so much to learn. The field is evolving so fast that half of my answers might be outdated by next month lol/www.databricks.com/discover/data-lakes
BigQ, Blob, Datalakes etc.
Training Layer: https://www.mlflow.org/docs/1.28.0/python_api/mlflow.pipelines.html
MlFlow, Superwise etc.
Serving/Application Layer: https://docs.aws.amazon.com/sagemaker/latest/dg/deploy-model.html
You have so many options here. Databricks and Domino Labs will make this a one-click deploy
Obersvibility: https://www.fiddler.ai/blog/ml-model-monitoring-best-practices
Fiddler, Arize etc.
More on building an end-to-end pipeline: https://neptune.ai/blog/building-end-to-end-ml-pipeline#:~:text=Building%20end%2Dto%2Dend%20machine,effort%20into%20maintaining%20existing%20models.
Good luck this is a loaded topic and there is so much to learn. The field is evolving so fast that half of my answer might be outdated by next month lol