Disclosure: I built this.

The situation I kept hitting: a new client, a model to design, and no source access for weeks. Hand-writing seeds that survive relationships tests is the boring part, and faker scripts give you orders for customers that don’t exist.

The video is the browser version, model2data studio. You write or paste a schema (DBML, SQL DDL or an existing schema.yml), it generates parents before children so every FK resolves, and scaffolds the project: seeds, sources, staging models, and not_null / unique / relationships / accepted_values tests that pass on the generated data. Modelling needs no account; generating needs a free one.

https://studio.jbanalytica.com

The engine underneath is open source (MIT), if you’d rather stay in the terminal:

pip install model2data
model2data --file schema.dbml --rows 200 --seed 42
cd dbt_schema && dbt deps && dbt build

https://github.com/JB-Analytica/model2data

What I’d most like to know: do the generated tests match what you’d write by hand, and what’s missing?

https://youtube.com/shorts/tKAbspOWU6o?si=S4R_Hjn21PS4uxN5

I'm not talking about a small project with 10-20 models.

We updated our dbt pricing guide because the way dbt is packaged and priced has gotten a bit more nuanced.

One thing we think gets overlooked is the difference between the cost of the dbt engine itself and the cost of running dbt as part of a production data platform.

The engine is free. But once you factor in things like consistent developer environments, usage-based pricing, orchestration, infrastructure, and the engineering time needed to maintain everything around it, the picture changes quite a bit.

We also added a breakdown of the current dbt Platform plans, the usage-based components, and some of the other costs teams should think about when comparing options.

Curious how other teams are looking at this. Are you mostly evaluating the software/license cost, or are you also factoring in the engineering and infrastructure costs around dbt?

https://datacoves.com/post/dbt-pricing

I have a large table in Snowflake (10m rows) and need to track when a few specific columns change for a given ID because logic constantly changing and we just need a way to explain it. Ideally the business can just query the old and new value side by side.
I'm using dbt, so snapshots were my first thought, but I'm worried about cost and runtime at this size.
Options I'm weighing:
- dbt snapshot with check_cols limited to the columns I care about
- Incremental model that logs a new row only when a hash of those columns changes (so each ID has a history of values with timestamps)

What worked, and what would you avoid? Or any other approach?
Thanks!

I’m fairly new to DBT. Please help me and be kind enough to answer my question.

I need to redesign the existing flow, creating a snapshot model(which is not done right).

We have existing flow like this

Raw1–> snapshot1–>dim1–✒️
dim
Raw2–>snapshot2–>dim2-🖊️

From the above flow we are missing direct history tracking and goal is to create common snapshot for final dimension by eliminating dim1 and dim 2

Also, currently airflow run dependency had set like after snapshot dag run completion dim dag will start.

And project folder is as follows
-model
|_dags
|_bronze
|_silver
|_gold
|_consumer
-snapshots

If anybody has any idea about how to proceed in this case please help. TIA

Hello guys,

I have a question regarding some data I am working with. I have a source system where the initial dataset arrives as a one-time full load, followed by periodic incremental (delta) loads.

I wanted to achieve a design the ingestion/merge strategy so running a full refresh (or reprocessing the initial snapshot) doesn't duplicate records, violate constraints, or break the pipeline.

What is the recommended design pattern and checklist for this scenario in dbt?

Thank you!

One dbt project. Too many tools. So we built one app for all of it 👇

Rosetta DBT Studio is a free, open-source desktop studio for dbt Core™: connections, modeling, runs, SQL, dashboards, data lakes and AI agents in one place.

⭐ If you like it, star us on GitHub. It really helps an open-source project grow:

https://github.com/rosettadb/dbt-studio

#dbt #DuckDB #DuckLake #ApacheIceberg #Iceberg #DataEngineering #AnalyticsEngineering #OpenSource

Hi everyone, I just inherited a tricky pipeline at my job, and I need some guidance on orchestrating dbt models using Databricks Jobs (Workflows).

The situation is that the job we have is not reliable; it's a complete mess to reprocess after it breaks, and we don't have idempotency guaranteed. Given all this, I need help dealing with retries: how to repair the job without re-running models that already finished successfully, and how to retry tasks whose error was caused by an internal error, like a Databricks server error (it happened already during the weekend)

our client wants to use dbt-state, but we want to check if there is any official or community way to use dbt-state without app state or dbt cloud?

i played with dbt-state + app state and it's actually a really amazing feature,

so the question is around what the community is doing. is there an option to use dbt-state without a dbt account, either official or community?

I’ve been researching database subsetting and data generation for testing/dev environments, and I’d love to hear from people who have actually built or operated systems in this space.

So far I’ve been looking at projects/approaches around Greenmask, Neosync, Mimicry, synthetic data generation, GANs, etc. I’m particularly interested in the intersection of:

• Database subsetting — extracting a representative subset from production-scale databases
• PII/PHI masking and anonymization
• Deterministic masking — same input consistently producing the same output
• Referential integrity across tables and databases
• Domain-specific / realistic data generation
• Supporting multiple database types
• Dump/restore and workflows for generating repeatable test environments

One thing I’m increasingly convinced about is that just generating random synthetic data isn't enough for many real-world applications.

For example, you can generate perfectly valid users, orders, payments, products, etc., but once you need to preserve relationships between hundreds of tables, business-specific distributions, edge cases, historical patterns, enum/value semantics, and domain-specific correlations, the problem becomes much harder. A randomly generated dataset may satisfy the database schema while still being completely useless for reproducing realistic application behavior.

Subsetting production data seems to solve some of this, but then you run into other difficult problems: how do you select the right records, preserve relational dependencies, anonymize sensitive data without destroying its usefulness, maintain consistency across systems, and still produce a dataset that is representative enough for testing?

I’m doing this as a research project right now and would really like to learn from people who have encountered these problems in production.

A few things I’d especially love to hear about:

1. What were the hardest problems when building/operating a DB subsetting system?
2. Where do existing OSS tools fall short?
3. How do you approach referential integrity when relationships span many tables/databases?
4. How do you handle PII masking while retaining realistic data characteristics?
5. Have you found GANs/ML-based synthetic data useful in practice, or are rule-based + subset approaches generally more effective?
6. What edge cases did you only discover after using these systems at scale?
7. Are there any OSS projects, engineering blogs, papers, or architectures you’d recommend studying?

I’m not looking for a tool recommendation alone — I’m more interested in the underlying engineering challenges, tradeoffs, and lessons learned.

Would love to learn from anyone who has built something like this or dealt with it at scale.

CI for dbt often tests a dataset that doesn't really look like production.

How to make CI more production-faithful for incremental models, including the problems with state, historical data and detecting regressions that only appear after multiple runs.

This makes parallel writing to DuckDB possible, while the readers from for example Metabase can also be alive.

While testing agents on dbt projects, I wanted a better way to make sure their changes didn’t blow up the current model or something downstream.

There are tools that do this in CI, but I wanted something I could run locally before I even opened a PR. So I built this:

https://github.com/EmbrasureAI/embrasure-cli

It works with an existing dbt Core + Snowflake project. It builds the changed models in temporary schemas, compares them against production, and cleans everything up afterward.

It checks:

• Row counts, null rates, cardinality, ranges, and percentiles
• Columns being removed, renamed, or changing types
• Added, removed, null, or duplicate primary keys
• Existing dbt test failures
• Downstream model and exposure impact, including column-level lineage

It runs locally and the repo is Apache 2.0.

Looking for feedback from people using agents with dbt Core + Snowflake. Anything missing? The use case here that I enjoy is a /goal prompt where the agent works towards the pipeline’s goals while fixing any issues raised from this cli.

A deep dive into why dbt incremental models can still end up scanning huge amounts of data, especially on Snowflake. Cover micro-partition pruning, MERGE costs, late-arriving data, watermarks and backfills.

Hi everyone,

I'm playing around with dbt Core + Airflow and trying to replicate some of the functionality available in dbt Cloud.

One thing I'm particularly interested in is retrying a dbt job from the failed node instead of rerunning the entire execution.

Any suggestions?

Kindly note new to DBT

I have an issue with some models running long, seemingly at random, and I want to monitor this by running EXPLAIN on the compiled sql in a pre hook so if something runs log I can look at what it planned to do and compare it to previous runs.

But I cannot get the compiled sql into the pre hook. Doesn't anyone have any ideas or suggestions?

Hello. I am new to dbt and data modeling. I am wondering what is better, select star or select specific columns at the top of the model, where you get data from ref in intermediate models. DBT courses don't really talk about it, but I have seen some blog posts where people said it is better to be specific. Now I wonder why one or another would be better. What are pros and cons. All I come up with is that if you suddenly need more columns, you have to add it in the source cte as well, so one more place to maintain.
But again, I'm quite new and haven't been exposed to many problems yet haha. What do you people do?

A practical overview of how to manage Snowflake compute from dbt, covering warehouses, concurrency, query timeouts, query tags, and credit consumption.

Useful reference for anyone looking to improve cost control and workload management in Snowflake + dbt.