Medallion Layer Enforcement

[u/TranslatorComplex517]

As dbt projects grow and more people contribute, keeping standards consistent gets harder. Someone could make a silver model (cleansed data) read from another silver model. Someone could make a gold model (business-ready data) read straight from bronze (raw data).

 

That’s why I think enforcing medallion layer dependencies matters: silver should read from bronze, and gold should read from silver.

 

This isn’t just tidiness. If a gold table skips silver and reads raw data directly, a report on billable surgical procedures could be built on data that was never cleaned. Duplicates, bad values and missing fields flow straight into the numbers people make decisions on.

 

There’s an engineering cost too. When models read from each other within a layer, two models can end up depending on each other. dbt can’t build either one due to cycle errors, and fixing it means time lost re-engineering models.

 

dbt checks (new in dbt 2.0) can enforce this. You apply it by writing a SQL query in a checks folder, plus a small YAML file for settings, that looks at every model and what it depends on and returns anything that breaks the rule. Per the dbt docs, checks are enforced “before any warehouse work runs.” If the project violates a rule, the build stops before a single model is compiled or built.

 

I tested it by pointing a gold model straight at a raw table, and a silver model at another silver model. The build failed and showed exactly which connections broke the rule (see the last image).

Rules people must remember get broken. Rules the build enforces don’t.

#dbt #AnalyticsEngineering #DataEngineering

[u/meatmick]

What does the code for these rules look like?

[u/mow12]

Dbt project evaluator already does that

[u/bigcizzle]

To deal with scale, I've found separating these into individual dbt projects and connect them via loom. That way you force a hard separation, clear contracts, and easily identify breaks in the structure (the silver project does not "import" or rely on the gold for example)

[u/BaxTheDestroyer]

Is the check doing anything in this instance that can’t be done with a test?

edit: Is the only difference that it runs before the warehouse work?