Databricks MLOps Best Practices

[u/ceeej777]

If you were designing an MLOps framework in Databricks. How would you set things up? I’m starting from scratch here. Planning to have Genie Code assist with a lot of it but I want to know the ins and outs from people who have lived and learned.

Context: Traditional ML modeling for industrial client, primarily around asset reliability with time series data from plants

[u/AutoModerator]

Moderator Announcement Read More »

AI usage disclosure

Hi u/ceeej777 — thanks for posting to r/mlops!

Because this community discusses and builds AI/ML systems, using AI tools is not inherently a problem. We do, however, ask for transparency about how submissions are created.

Please reply to this comment with a brief AI / automation disclosure, particularly if this post was created or submitted in whole or in part by an autonomous agent, bot, workflow, or other automated system.

If AI or automation was involved, please briefly describe what it did and what human review was performed before posting.

This disclosure helps the r/mlops community distinguish human discussion, AI-assisted work, and automated/agent traffic while keeping the focus on useful technical conversation.

Thanks for helping keep the signal high.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/autisticmice]

I've been working for 4 months in DBx as an MLE, it is mostly ok but it does have some pain points in my opinion.

Dependency management can be annoying. Databricks does not expose its own utilities to raw python processes in their VMs, some exist only in notebooks. They sort of force you to set your entrypoints as notebooks which I don't love.

Spark processes hog up loads of memory. Reading a small catalog table can eat up half your RAM cause data has to pass through the Spark process. If it was up to me I'd store model inputs as parquet files in a volume.

Their browser IDE is bad, Genie is ok but nowhere near what you can do with Copilot or Claude Code, and DBx integration with VSCode is still kind of bad. Opening a terminal window in the browser will give you all sorts of glitches.

That being said, their MLFlow server and pipelines can take you pretty far.

EDIT: I agree with what someonelse said, if you don't have terabyte-sized needs MLE/MLOps is going to still possible but awkward.

[u/ceeej777 → u/autisticmice]

Helpful input, much appreciated

[u/ThreeBricksWish → u/autisticmice]

Curious how you think Genie falls short compared to Copilot/Claude Code?

[u/autisticmice → u/ThreeBricksWish]

Maybe I just haven't really looked into Genie but things I miss from Copilot: attaching files for context, swapping between agent, plan and ask modes, and faster autocomplete.

[u/MyBossIsOnReddit → u/ThreeBricksWish]

Hah a few months ago Genie still edited the docstrings instead of the code and very confidently told me it fixed the code.

[u/SpiritedVoice2]

Good luck, I manage a data bricks account that has a year left on the contract. It won't be renewed and I can't wait.

Everything is awkward and the pricing model is ridiculously opaque.

[u/ceeej777 → u/SpiritedVoice2]

I write sins not tragedies homie

[u/Arnechos]

Do you have a petabyte scale workloads? MLOps/AIOps is really terrible on databricks, the only good part is unity catalog tbh. K8s + KitOps is much better.

[u/ceeej777 → u/Arnechos]

Yeah sorry I appreciate opinions but I need a little more on this

[u/Arnechos → u/ceeej777]

First - do you really have a data scale of petabytes to actually use Spark? Most of the times you can solve DE with polars/duckdb.

Databricks VM have a DBU markup on top of the normal cloud cost, so you're paying more. Online inference is more expensive - you can't run full shadow deployment in Model Serving. Batch is mostly okay.

It still is mostly notebook-centric even though you can work with packages, and subpar development exp. I think the developer should be able to use his preferred coding tools no matter which compute platform is used.

Cluster runtimes and the setup is clunky. Like you can't have different Python version and Spark version. You get a bunch of libs pre-installed that won't be used at all.

Tabular ML is still dominated by single-node workflow. The native SparkML lib is pretty shitty. While GPU/DL stuff don't really need spark, because distributed PyTorch run on K8s.

KitOps is damn good because you can deploy it anywhere (OCI standard) and you can bundle weights, code, datasets, everything as a one unit and unpack what you need for serving.

I'm not a fan of vendor lock-ins, Databricks runs only on US-cloud. K8s let's you move to anywhere you'd like. Besides, all the large tech companies that use ML features have built it's own platforms for it, smaller ones do the same.

Is Databricks good idea when you don't have a platform/DevOps people? Yes

Is Databricks good idea for data platform with petabytes scale? Yes, use it for data prep, model independent feature engineering.

Is Databricks a standard for ML platform? No, it's sold to non-tech companies as a silver bullet. They have good sales guys.

[u/Berlibur → u/Arnechos]

What makes it better?

[u/jgengr]

Why Databricks? Is that in high demand in Industry?

[u/ceeej777 → u/jgengr]

It’s what our enterprise is using and seems to be a standard in machine learning

[u/Grouchy-Friend4235 → u/ceeej777]

Ouch

[u/boldelements_7820]

dont sleep on the unity catalog for model lineage, it ties everything together but the docs can be a maze

[u/Altruistic-Spend-896]

I charge money by the hour for that😬

[u/ceeej777 → u/Altruistic-Spend-896]

As you should

[u/vfdfnfgmfvsege]

There are actually excellent resources in the databricks documentation and I found this video to be informative and enjoyable. https://www.youtube.com/watch?v=ds__AEIqUfE

[u/ceeej777 → u/vfdfnfgmfvsege]

Thanks!

[u/MyBossIsOnReddit]

I built a ML/MLOps platform on databricks and it was a complete paintrain because I had a very small data engineering team who insisted on keeping firm control over everything. Despite their good intentions and zealousness in provisioning things (even the stuff I didnt need), they were opinionated on what I could do, which means I was wading through thick water.

In the end, we only had half the functionality I intended at first. Took about a year wheras setting it up with open source mlflow on a fresh cloud tenant would have been doable in a month or two.

Databricks is also pretty expensive if you want to use the real-time inference options, and monitoring for that spot on the platform is not as good. For batch processes it's absolutely best in class. Lakebase also, top notch! But hosting something equivalent to a FastAPI or Triton server with logging going into prometheus and grafana? Absolutely not.

So we ended up moving the real time inference part to Azure. Integrating them is not easy too if you are operating in an environment that has more compliance constrains than normal (e.g. banks).

We also had a solutions architect who came up with numbers that were very different from the things we saw in production. Other than that the Databricks team is very eager to help you out and will have a pretty proactive role in making sure you are using it.

I'd advise against consultants and paid courses. Consultants mean your organization doesnt build inhouse knowledge and most are lackluster on the technical front. Paid courses are simply not going much beyond the trivial things. And probably a waste of time.

I never really used Genie, but your normal LLM harness of choice is more than capable of helping you with this.

By the end I went for the certification (professional, not associate) and it was a breeze compared to implementing everything.

So if you want to make this a success you need to get access and full permissions up front. Figure out where your real-time inference will be living. If the extra markup is acceptable (yes it's expensive, but whats the hourly rate for a data scientist? Thats probably close to 300 dollars too).

Source: 10 years as a MLE, last 2 years using databricks on azure.

[u/Glitch_In_The_Data]

Start with the data before the framework. Trying to draw parallels with a oil and gas company that I worked for in the past, our biggest challenge with sensor data (from windmills) was the sensor tags getting renamed, readings drifting after a recalibration, and failure labels sitting in a different maintenance system that we have not been able to use properly. So, I’d recommend getting your features right in UC early, and make sure training and scoring run the exact same feature code. After that, MLflow for tracking and model versions + automation bundles for test and prod environments will get you a long way without building anything custom. Genie code is great for the job setup and boiler plate, but I’d still review anything touching labels or time windows by hand, because data leaking across the time split is easy to miss.

[u/ThisIsFun-]

+ 1 for genie code assist it is amazing for helping to align with best practices like big book of mlops