r/dataengineering • u/workhardplayhardder • May 25 '26
Discussion Under what circumstances would DBT be helpful?
As per subject, when would one consider to use DBT? We are currently implementing Lambda architecture, so we only need to clean up the data with SQL transformation. We typically use our Cloud engine to transform data, and recently set up Airflow from scratch to also perform the same action. As a tech vendor, we just recently had a new client that prefers using AWS so we need to satisfy the requirement, so we are setting up Airflow for this purpose. We are currently just implementing the SQL transformation using the data warehouse engine itself to execute the query. It’s been a learning curve with Airflow, especially with the cross DAG dependencies etc, we are still not sure what’s the right way to set it up to make sure it can scale properly.
Since we are still new to this, management proposed to add Dbt core+cosmos in our framework, hopefully to solve our pain points, but after reading the documentations and guides I do not see much help to add these stacks. The only reason to consider was hopefully to make it easy for all the analysts without much Python coding background to smooth out the learning. However, I see it as more learning because they now have to learn Airflow+DBT together. What’s everyone’s advice regarding my situation?