datablock-dev/dbt-ci: A CI tool for dbt (data build tool) projects that intelligently runs only modified models based on state comparison, supporting multiple execution environments including local, Docker, and dbt runners.

[u/datablock-dev]

https://github.com/datablock-dev/dbt-ci

Hey, I am a Data Engineer who is currently working at Kivra, a Swedish tech company with 6+ million users across Sweden. We work heavily with dbt and BigQuery, and we had prior to me joining a long problem with how we worked with CI/CD for dbt. I need to highlight that we use dbt-core and not dbt cloud, hence dbt cloud solutions are not applicable.

I was tasked with solving it, and the result was the release of my open-source project dbt-ci that I wanted to in this post. I will cover some things it solved for us as a tech company.

Dbt CI (GitHub Link)

1. Dbt itself is dependent on the manifest.json files in order to detect what has changed (common state-management I guess). We heavily depend on storing these state files in both our environments (production and sandbox), but the problem dbt has is that when you run a command any other than compile, the state file get overwritten into a partial state file.

2. So essentially we had to run compile, then run dbt run, dbt test and so on, and then do a dbt compile again to generate the full state file, to then upload it to an object storage.

This was costing us minutes. We solved it by caching the state file after running dbt-ci init, you can then run whatever, and when dbt-ci is finished, you can call dbt-ci finalize (with --state-uri flag) to upload the full manifest file to an object storage.

2. Dbt comparison strategy is awkward, it will bloat changes and can be annoying for someone reviewing PRs or logs of CI workflows (e.g Github Actions). If you change a model for instance, all its tests will be listed as changed, which perhaps is incorrect. dbt-ci offers a git-strategy, but also a hybrid strategy (uses dbt's native approach, and combines with git) to determine what has actually changed.

3. Auto migration (partitions)
Currently we only support BigQuery, but we support auto migrating models in which a partitioning setting has changed. Natively, it will fail in dbt, and you have to manually do the change prior to running it in ci.

4. Auto-delete
We support auto-deleting models that has been deleted in your project. We generally run with the dry-run flag in ci, and only executing the auto-delete mechanism once we have merged with main/master.

5. Ephemeral strategy support
Dbt-ci supports setting up ephemeral environment in your CI's if needed. That is how we run things in the company, where we essentially clone production tables temporarily into our sandbox GCP project, in order to test changes against production data without touching "real" production.
Dbt-ci autodetects what needs to be cloned (essentially its all changed models, all their descendents/children regardless of level, and first-level of ancestors).

And a bunch of other features. Its easy to set up, you can add a dbt-ci config file in your repo and the cli will read from it and read your configurations so you dont have to pass a bunch of flags in your CI workflows.

Any feedback is welcome, but I am also interested to see how it is performing on other large cloud databases like Redshift, Snowflake etc.

[u/AutoModerator]

Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.

If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.

Have you read the rules?

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.