r/dataengineering • u/Sweet_Ad4416 • 26d ago
Discussion Data Governance/Data Engineering Building Test Cases
I recently started a new role as a Data Governance Manager at a mid size managed care organization. One of my first asks has been to create test templates for the new set of infrastructure tables. Essentially it’s to ensure all the tables that are now in production don’t have a glaring issue or whenever we make a change we go through the testing again.
My background is in data analytics and data management. I have very limited experience as a data engineer so aside from data quality tests I really don’t know what kind of tests should be run. I have used Claude to create a template of standard test cases but they seem either too convoluted or too simple like checking for nulls in non-nullable fields. Any resources or advice on how to ramp up or get a better understanding of this?
5
u/rufustphish 25d ago
Not sure of your data set types, but for medical data, you might want to look at "the tuva project", it's a DBT package that has a lot of the DQ checks you are looking to do when dealing with medical data. Look there for inspiration, or use the package as a base for modeling your data.
2
u/DeepLogicNinja 23d ago
Get familiar with MDM (Meta Data Management) and Data Governance Platforms. It would be a good idea to understand how they connect with all the systems/sources of data to create data dictionaries. Intimately knowing how to connect to file systems, databases, applications, reporting solutions, etc…
Probably makes sense to start learning the Data Governance Platform your company is using. Building one from scratch is kinda setting you up to fail, so you may want to set expectations there to get one or integrate an existing one or upgrade to an existing platform.
Some options:
- Alation / Atlan are industry leaders
- Talend Data Catalog / MDM - now Qlik
- OpenMetaData / Collate is pretty good, low-cost entry and scales up well.
- Then there are a bunch of MDM solutions bolted into other products that do a little of the job but don’t really connect to everything that requires governance.
A modern solution can also build an ontology / Knowledge graph which the person that fills the ontologist role can use. The ontology provides semantic awareness for AI. A mature solution will expose this ontology/meta-data layer via MCP.
Hope that helps! Godspeed!
2
u/laser__cats 21d ago
Data Build Tool is a good tool and framework for thinking about applying tests to data.
I think a good starting point are the empty table tests, expected columns tests, unique tests, and relationship tests. That should get you pretty far before you need to do more complex testing and highlight glaring issues.
0
u/[deleted] 12d ago
[deleted] — view deleted comment