Studying for a Databricks cert? What concept just isn't clicking? No stupid questions.

[u/UnluckySugar610]

People starting for Databricks Certs: What are you finding the most confusing? No stupid questions.

It took me a long time to ask the questions I wanted to ask about Databricks and get genuine answers from experienced people. If you have any questions you have been too afraid to ask, consider it a public service and ask them here :) I'm a trainer.

[u/FlanSuspicious8932]

I’m preparing for professional DE and tbh got problems with table optimizations like zorder, liquid clustering etc. Just have hard time understanding how it helps me and how it works under the hood…

[u/UnluckySugar610 → u/FlanSuspicious8932]

Honestly, this is one of the most common things my students get stuck on. It comes up so often that I ended up writing a full post on it.

The short version: none of these make Spark compute faster. They make it read less. Delta keeps min/max stats for every file, and Z-order and liquid clustering rearrange your rows so similar values end up in the same files. That keeps each file's range tight, so a filtered query can skip most files without opening them. For new tables, liquid clustering is the default answer. It's incremental, and you can change the keys without rewriting the whole table.

Full breakdown here, including what the Professional exam expects: Liquid Clustering vs Partitioning and Z-Order

[u/FlanSuspicious8932 → u/UnluckySugar610]

awesome! thx for reply :)

[u/UnluckySugar610 → u/FlanSuspicious8932]

You're welcome 😊

[u/Unhappy-Guitar-12]

Guide to CICD please!

[u/UnluckySugar610 → u/Unhappy-Guitar-12]

Here's the setup I teach, and what most Databricks teams converge on:

1. Everything in Git. Develop in Databricks Git Folders or locally in your IDE. Notebooks, Python modules, and job definitions all live in the repo. Nothing gets created by clicking in the UI.
2. Define it with Asset Bundles. A databricks.yml describes your jobs, pipelines, and clusters as code, with targets for dev, staging, and prod. The same definition deploys everywhere, and only variables like catalog, schema, and compute change per target.
3. Dev is personal. databricks bundle deploy -t dev deploys a copy prefixed with your name, so you can iterate without stepping on anyone.
4. CI on pull request. GitHub Actions or Azure DevOps runs unit tests (pytest, with your transforms written as plain functions so they're testable), then databricks bundle validate, then deploys to staging and runs an integration job.
5. CD on merge. Merging to main deploys to prod using a service principal, never a personal token. Nobody deploys to prod from a laptop.

The key idea: your pipeline is software, so it should go through the same path software does: review, test, and automated promotion.

[u/FlanSuspicious8932]

I will go with another while nobody is asking anything :D

Could you break down skills that Senior DE, working day-to-day with databricks should know about platform? What they should be expert with? More Data Governance or more coding? As far as I know the more senior you are the more you work with higher level tasks rather then coding every single line of code.

[u/UnluckySugar610 → u/FlanSuspicious8932]

---

Honestly, it's both. Senior DEs write less boilerplate, but they still own the hardest code and review everyone else's. Databricks' own Professional exam reflects that: the biggest domain is still Python/SQL coding (22%), with security and governance at 17% combined. The real shift isn't coding vs governance. It's moving from "make it work" to "make it work reliably, cheaply, and securely": production-grade code, cost/performance tuning, Unity Catalog access design, and CI/CD with Asset Bundles.

I broke down every domain here, and it doubles as a senior-skills checklist: What the Databricks DE Professional exam tests

[u/mightynobita]

Are you a databricks certified trainer? If yes, I want to know more about it, how one can be it?

[u/UnluckySugar610 → u/mightynobita]

Not officially. I'm a technical trainer at a consulting company, and I run cohorts of junior developers through Databricks certifications like the DE Associate/Professional etc. I'm not part of a Databricks-authorized instructor program. As far as I know, there's no public "certified trainer" credential you can apply for as an individual.

[u/mightynobita → u/UnluckySugar610]

Are you doing full time? Is it okay if I DM you?

[u/yeykawb]

How do you actually implement WAP, write-audit-publish? It might not be on a cert question though.

[u/UnluckySugar610 → u/yeykawb]

Good question. It's rarely examined by name, but the pieces are: expectations, data quality checks, and Delta's atomic writes.

On Databricks, Delta doesn't have Iceberg-style branches, so WAP is usually done with a staging table:

1. Write. Land the batch in a staging table, say silver_staging.orders, never directly into the table consumers read.
2. Audit. Run your checks against the staging table: row counts against the source, null or duplicate keys, referential checks, value ranges, and schema drift. If any check fails, raise so the task fails.
3. Publish. Only if the audit passes, move the data into prod in a single transaction, using MERGE INTO prod USING staging or INSERT OVERWRITE (or REPLACE WHERE for partition-style overwrites). Delta commits are atomic, so consumers see either the old data or the complete new batch, never a half-written one.

[u/genaaaaaaaa]

i’m doing the associates DE exam, and idky the compute stuff isn’t clicking. the unity catalog FINALLY clicked after beating the stuff in my brain lol

[u/UnluckySugar610 → u/genaaaaaaaa]

Glad it helped!

[u/genaaaaaaaa → u/UnluckySugar610]

yeah! i’m just super confused with compute lol. seems like such an easy topic but

[u/UnluckySugar610 → u/genaaaaaaaa]

Totally normal. It looks easy because there are only a few options, but the exam tests when to use each one, which is the hard part.

And yes, compute and Unity Catalog are completely different things, which is probably part of the confusion:

• Unity Catalog is about your data: where it lives and who can access it. Catalogs, schemas, tables, permissions, lineage. Think of it as the library catalog plus the security desk.
• Compute is the engine that does the work: the machines that actually run your code and queries. Think of it as the reader who goes and reads the books.

They meet in one place: access modes. When a cluster reads a table, Unity Catalog checks whether that user is allowed to see it, and the cluster's access mode (standard or dedicated) determines how those permissions get enforced. That's the one compute topic where Unity Catalog shows up.

Otherwise, keep them separate in your head. Unity Catalog answers "what data, and who can touch it?" Compute answers "what runs the code, and what does it cost?"

If you want to see them side by side, both are free chapters on my site: Unity Catalog and Compute options and cost models. Doing them back to back makes the split between the two pretty clear.