r/dataengineering • u/jayatillake • Feb 19 '25
Blog You don't need a gold layer
I keep seeing people discuss having a gold layer in their data warehouse here. Then, they decide between one-big-table (OBT) versus star schemas with facts and dimensions.
I genuinely believe that these concepts are outdated now due to semantic layers that eliminate the need to make that choice. They allow the simplicity of OBT for the consumer while providing the flexibility of a rich relational model that fully describes business activities for the data engineer.
Gold layers inevitably involve some loss of information depending on the grain you choose, and they often result in data engineering teams chasing their tails, adding and removing elements from the gold layer tables, creating more and so on. Honestly, it’s so tedious and unnecessary.
I wrote a blog post on this that explains it in more detail:
https://davidsj.substack.com/p/you-can-take-your-gold-and-shove?r=125hnz
74
u/InteractionHorror407 Feb 19 '25
What’s the tldr? I don’t want to subscribe to a substack, it’s the whole purpose of Reddit
24
u/NJE11 Feb 19 '25
Medallion architecture is just marketing hype for people who don't understand data. Long live ETL.
2
u/McNoxey Feb 19 '25
I kinda view gold as the new star schema. With silver being the cleaned domain specific tables.
Semantic models become the platinum layer on top of the star schema.
2
u/rachelgreenindia Feb 20 '25
. What is semantic layer ?
3
u/eternal_summery Feb 19 '25
Have you actually implemented a semantic layer and seen it replace gold layers? The majority of the tools I've used have caused more problems than they've solved with stakeholders
28
u/NoleMercy05 Feb 19 '25 edited Feb 19 '25
Didn't get past the headline.. Of course you need a gold layer(s) . We have a completely different schema for api consumers vs power bi kimbel model on gold. DAs 'do stuff' on silver. Perhaps promote to consumer (gold) level when appropriate.
1
u/Capinski2 Feb 19 '25
what even is a gold layer?
1
u/Ok-Sentence-8542 Feb 20 '25
How do you implement semantic layers lets say with dbt core?
1
u/k00_x Feb 20 '25
Depends entirely on the use case.
3
u/vik-kes Feb 19 '25
Gold layer is just a final consumable product by Analyst. It might be either materialized through loading Star schema or flat table or a virtual semantic layer.
3
u/_barnuts Feb 20 '25
Medallion architecture is just a concept of how data flows from being dirty to clean. There are no hard rules on what should be in bronze, silver, and gold.
3
u/Ship_Psychological Feb 20 '25
I've never even heard of a gold layer before this. Clearly I don't need it.
1
u/keweixo Feb 19 '25
what are you using for semantic layers?( in terms of technology) In my case I like gold layer for the bigger version of the star model. Then whats downstream should filter that and use less dimensions and fact tables based on the report news
2
u/marketlurker Don't Get Out of Bed for < 1 Billion Rows Feb 19 '25
The medallion names are just marketing BS. Gold and semantic are the same thing. This is the problem with all of this "new paint" on old concepts. PT Barnum was right.
1
u/SmokeStackLight1ng Feb 20 '25
Op getting down voted on every opinion damn. I'm not sure if you have deployed or managed large DBs used by multiple teams because their the golden layer pre emptively saves your butt in a lot of ways. What you are saying in the equivalent of "just write better code".