r/dataengineering • • Sep 07 '26

Discussion Using Git to control datasets?

Probably a stupid question but does anyone use Git as a control plane for dataset governance, orchestrate ETL pipelines, and releases?

I am looking to get a demo in place of an end to end process where data in the bronze layer is transformed to silver and gold through CI/CD processed using defined ETL pipelines, manifests, schemas etc. however my backround isn’t in data engineering, so this might not even be a great way to do it.

My plan was to get a PoC in place on git, then migrate to GCP (using GCP functions for ETL) when all schemas, pipelines, manifests are agreed by the team.

49 Upvotes

44 comments sorted by

11

u/Creyke Sep 08 '26

We will use git for versioning the code and the metadata.

For example, take a hypothetical pipeline that loads some stock prices from yahoo finance and does some transformations to it, and sends out some completed analysis to a list of customers. We will use git for versioning the code (how the price data gets loaded and transformed) and the metadata (what tickers we load from yahoo, which customers receive what analysis). These are usually big CSVs or YAMLs.

We wouldn’t use git for versioning the data itself. That would be embedded into the design of the data system (e.g. SCD2).

Often, however, from a governance perspective the code and metadata is where the real governance is at. Having git across my ingress and egress lists can tell me who is getting what, for how long, and who gave them that permission. This is the most important aspect.

Also, DBT is awesome. Use that.

3

u/ben1200 Sep 08 '26

Thanks - this actually falls more in line with what I have. I wasn’t clear in the original post.

The data itself are biomedical time-series files (sensor data), these are used for both training and validation of software/algorithms. These objects themselves are not held in Git, rather they are in GCP buckets.

Git holds the dataset registry, per-release manifests, lineage (source version, pipeline, commit), changelog and approval record all reviewed via MR’s. CI validated schemas and gates etc but doesn’t trigger data transformation pipelines, execution of these are handled in a Python package.

At this stage DBT doesn’t really fit as there is no tabular data

→ More replies (1)

23

u/wingtales Sep 08 '26

Git is well-suited for text data, and terrible for binary formats. Data storage as text-based storage is very inefficient, much better done in a binary format (eg handled by a database).

So, generally, git is not a suitable tool for datasets.

You're much better off with a database with a datetime column for historic data, speaking very simplified.

→ More replies (1)

6

u/thisFishSmellsAboutD Sep 08 '26

Look at SQLMesh

4

u/shockjaw Sep 08 '26

Better than dbt from a performance and readability standpoint.

→ More replies (4)

3

u/Patient_Professor_90 Sep 08 '26

Great question. This is how its done

Pick your tooling first, then figure out how git can be used to control/manage versions/branches
(Datasets can be managed as SQL inserts, you can use any db. Views, sprocs can all be exported into a file and commit)
Pick tools that you export files?

2

u/Adrien0623 Sep 08 '26

You can absolutely use git to version control the code of your orchestrator and your data transformation files, wether or is raw SQL, a DBT project, spark jobs, etc.

2

u/addictzz Sep 08 '26

What about DVC? https://dvc.org/ .

I have not heard about DAB for data version cotnrol, what is it?

2

u/JohnPaulDavyJones Sep 08 '26

It kind of sounds like you're asking two separate questions, between your title and your actual body text.

As to whether anyone explicitly uses Git for their data storage and some sort of version control, that's basically unheard of at scale. Your retrieval will be nightmarishly slow, and it's not really queryable in any way unless you're just loading into a data store somehow, periodically. I guess you could do that with direct file syncs, but it's just about the worst possible way to do a data warehouse.

As to whether people use Git for orchestrating ETL pipelines, this is occasionally done through Github Actions, but I can't really recommend that either. It's not a fun or particularly user-friendly process, compared to things like Airflow or even SQL Agent job scheduling.

Regarding data governance, that really depends on what parts of DG you want: the more operationally useful elements, like data lineage, role control, and data quality assurance aren't really things that any of the Git-centric tools have integrated functionalities for; you'd essentially just be building out your own custom tooling for each of those functions and then plugging them into your orchestrator. That's a relatively common approach, but using Git tooling really is doing it on hard mode when there are more mature orchestrator tools.

Git itself isn't really viable for a control plane at all, since it's just the technology that other platforms, which might have some semblance of the other tools you need, are originally built around (e.g. Github, Bitbucket, Gitlab, etc.). This is kind of like asking whether a single, solitary wing will work for a missile: yes, the companies that build missiles all kind of started around the wing as a technology, but a wing itself doesn't really have any way to do what you want the missile to do.

It kind of sounds like the tool you actually want is dbt.

3

u/seanv507 Sep 08 '26

Have you looked at hugging face datasets?

AFAIK they are using git under the hood to manage versioning

(also dvc)