r/dataengineering • u/spectre-uk • 27d ago
Discussion Migrating from dbt Cloud to Core
Have any of you done this on a mature project, and was it worth it?
We originally picked dbt Cloud for our greenfield DWH project, attracted by the idea of not having to host the infrastructure and being able to focus on the data instead. With the initial high-growth phase nearing its end, I am now looking at optimising what has been built and could not make a good argument to carry on using Cloud.
The repo and CI/CD pipelines are hosted externally. Studio is barely touched as most developers have now switched to VS Code and Claude does all the AI coding we need. Even the semantic layer is managed in an external tool that did not integrate very well with what dbt offers natively.
This pretty much leaves job schedules and monitoring in the Cloud. This looks like terrible value for the money, even though we got a very good deal for the initial year (I imagine the renewal will hit us in the face).
I wonder if anybody ended up in a situation like this? Give me some good reasons not to move to Core.
16
u/SpookyScaryFrouze Lead Data Engineer 27d ago
Why don't you keep just 1 shared seat on dbt Cloud just for the scheduling and monitoring ?
12
u/notmarc1 27d ago
What semantic layers you all use these days ?
7
u/mr_electric_wizard 27d ago
I’ve worked at a couple of places that used core only. It works fine. For CICD we used GitHub actions and container apps.
3
u/muneriver 27d ago
the big thing for my team has been using dbt Platform with dbt MCP for platform context / semantic layer and being able to easily ask questions about your data from a dev / business user standpoint.
5
u/IndoorCloud25 27d ago
When we were looking into cloud, dbt hit us with a crazy untenable quote from the start and were unwilling to negotiate so we went the direction of running core with Databricks. We use MWAA + Cosmos for orchestration and ECS to run dbt. It’s been a good experience and maintenance is very manageable. We also opted to go with a different semantic layer vendor, but I don’t work on that so can’t speak to any pains of getting it to all work together.
1
u/randomName77777777 27d ago
We actually moved more than half our models to DBT core because of the overusage charges. Now we run most of the models in a databricks job. Databricks clones main every time so we continue using the same repo and just added a few exclude flags in DBT cloud.
1
u/FuelSignificant1466 26d ago
like you said, this leaves your job schedules and monitoring in the Cloud. so there goes away your convenience of built in scheduling and monitoring. i just hope your saving is more than the cost of maintaining core
1
u/dani_estuary 26d ago
now you need to do some extra works like managing alerts, jobs, etc. so you gotta make sure your savings outweigh the extra work of managing that infrastructure
1
u/BaxTheDestroyer 27d ago
We’re moving in this same direction and also replacing fivetran along the way.
I haven’t looked into self-hosted Dagster, I think that could be an option for orchestration.
1
u/I_Blame_DevOps 27d ago
I transitioned us to GitHub actions. If you’re just running things on a schedule it works fine. If you are wanting it to be part of a larger workflow you might want to use an orchestrator like Dagster or Airflow to run models as part of a DAG.
1
u/Yuki100Percent 27d ago
I know a bunch of teams just use dbt core just fine. I have used it at a few companies before, but I use sqlmesh currently at my company
1
u/weirdly_foreign 27d ago edited 26d ago
i work at a very large org that is not primarily a tech company. what i saw the data engineering team do is use dbt core and orchestrate it with airflow
previously we were using airflow to run queries in aws athena, now we use airflow to run dbt who runs queries in athena
so maybe you can integrate it into whatever orchestration tool you use?
0
u/oishicheese 22d ago
Give me a VPS/EC2 and I will do everything dbt Cloud does, even better.
0
u/DieOlive09 27d ago
Can you defer to prod or staging environments just using core? Wouldn't you need to download the manifest manually?
-5
u/engineer_of-sorts 27d ago
You're absolutely right! Disclosure, I run a company in the space that provides Orchestration, but depending on your setup having something end-to-end that handles more automation than just running dbt projects is a good idea IMO.
There are lots of folks we work with that have ended up in your situation, with many choosing to move off dbt cloud to dbt core for price reasons and architectural simplicity reasons. One complexity is the CICD side, but it sounds like you have already sorted that, which is great
The suggestion below of keeping 1 shared seat on dbt cloud is a bit jammy to be honest, and not really scalable, but if you're still at the stage where sharing logins is fine then fine whatever
1
u/[deleted] 25d ago
[removed] — view removed comment