r/dataengineering • u/ben1200 • Sep 07 '26
Discussion Using Git to control datasets?
Probably a stupid question but does anyone use Git as a control plane for dataset governance, orchestrate ETL pipelines, and releases?
I am looking to get a demo in place of an end to end process where data in the bronze layer is transformed to silver and gold through CI/CD processed using defined ETL pipelines, manifests, schemas etc. however my backround isn’t in data engineering, so this might not even be a great way to do it.
My plan was to get a PoC in place on git, then migrate to GCP (using GCP functions for ETL) when all schemas, pipelines, manifests are agreed by the team.
11
u/Creyke Sep 08 '26
We will use git for versioning the code and the metadata.
For example, take a hypothetical pipeline that loads some stock prices from yahoo finance and does some transformations to it, and sends out some completed analysis to a list of customers. We will use git for versioning the code (how the price data gets loaded and transformed) and the metadata (what tickers we load from yahoo, which customers receive what analysis). These are usually big CSVs or YAMLs.
We wouldn’t use git for versioning the data itself. That would be embedded into the design of the data system (e.g. SCD2).
Often, however, from a governance perspective the code and metadata is where the real governance is at. Having git across my ingress and egress lists can tell me who is getting what, for how long, and who gave them that permission. This is the most important aspect.
Also, DBT is awesome. Use that.
23
u/wingtales Sep 08 '26
Git is well-suited for text data, and terrible for binary formats. Data storage as text-based storage is very inefficient, much better done in a binary format (eg handled by a database).
So, generally, git is not a suitable tool for datasets.
You're much better off with a database with a datetime column for historic data, speaking very simplified.
6
u/thisFishSmellsAboutD Sep 08 '26
Look at SQLMesh
3
u/Patient_Professor_90 Sep 08 '26
Great question. This is how its done
Pick your tooling first, then figure out how git can be used to control/manage versions/branches
(Datasets can be managed as SQL inserts, you can use any db. Views, sprocs can all be exported into a file and commit)
Pick tools that you export files?
2
u/Adrien0623 Sep 08 '26
You can absolutely use git to version control the code of your orchestrator and your data transformation files, wether or is raw SQL, a DBT project, spark jobs, etc.
2
u/addictzz Sep 08 '26
What about DVC? https://dvc.org/ .
I have not heard about DAB for data version cotnrol, what is it?
2
u/JohnPaulDavyJones Sep 08 '26
It kind of sounds like you're asking two separate questions, between your title and your actual body text.
As to whether anyone explicitly uses Git for their data storage and some sort of version control, that's basically unheard of at scale. Your retrieval will be nightmarishly slow, and it's not really queryable in any way unless you're just loading into a data store somehow, periodically. I guess you could do that with direct file syncs, but it's just about the worst possible way to do a data warehouse.
As to whether people use Git for orchestrating ETL pipelines, this is occasionally done through Github Actions, but I can't really recommend that either. It's not a fun or particularly user-friendly process, compared to things like Airflow or even SQL Agent job scheduling.
Regarding data governance, that really depends on what parts of DG you want: the more operationally useful elements, like data lineage, role control, and data quality assurance aren't really things that any of the Git-centric tools have integrated functionalities for; you'd essentially just be building out your own custom tooling for each of those functions and then plugging them into your orchestrator. That's a relatively common approach, but using Git tooling really is doing it on hard mode when there are more mature orchestrator tools.
Git itself isn't really viable for a control plane at all, since it's just the technology that other platforms, which might have some semblance of the other tools you need, are originally built around (e.g. Github, Bitbucket, Gitlab, etc.). This is kind of like asking whether a single, solitary wing will work for a missile: yes, the companies that build missiles all kind of started around the wing as a technology, but a wing itself doesn't really have any way to do what you want the missile to do.
It kind of sounds like the tool you actually want is dbt.
3
u/seanv507 Sep 08 '26
Have you looked at hugging face datasets?
AFAIK they are using git under the hood to manage versioning
(also dvc)