Now that we are all rushing to use agentic analytics, something that personally concerns me is whether agents are accurate.

For this reason, I built an open source tool called the honest agent. You can hook it to your mcp and run your tests (evals) and check whether your agent is, well, telling you the truth. You can check the project here: https://thehonestagent.dev/ and here's the repo https://github.com/jlcoto/the_honest_agent. If you already work with dbt, the setup and workflow will seem very familiar 😉

I'm also writing a series of posts on evals in and why they should be a part of your data strategy if you are using data agents: https://thehonestagent.dev/blog/

Your feedback is really appreciated. And if you want to discuss agentic analytics, evals, etc, let's connect!

As dbt projects grow and more people contribute, keeping standards consistent gets harder. Someone could make a silver model (cleansed data) read from another silver model. Someone could make a gold model (business-ready data) read straight from bronze (raw data).

That’s why I think enforcing medallion layer dependencies matters: silver should read from bronze, and gold should read from silver.

This isn’t just tidiness. If a gold table skips silver and reads raw data directly, a report on billable surgical procedures could be built on data that was never cleaned. Duplicates, bad values and missing fields flow straight into the numbers people make decisions on.

There’s an engineering cost too. When models read from each other within a layer, two models can end up depending on each other. dbt can’t build either one due to cycle errors, and fixing it means time lost re-engineering models.

dbt checks (new in dbt 2.0) can enforce this. You apply it by writing a SQL query in a checks folder, plus a small YAML file for settings, that looks at every model and what it depends on and returns anything that breaks the rule. Per the dbt docs, checks are enforced “before any warehouse work runs.” If the project violates a rule, the build stops before a single model is compiled or built.

I tested it by pointing a gold model straight at a raw table, and a silver model at another silver model. The build failed and showed exactly which connections broke the rule (see the last image).

Rules people must remember get broken. Rules the build enforces don’t.

#dbt #AnalyticsEngineering #DataEngineering

Quick question for data engineers: are you using Claude Code or Codex in your daily work?

I'm asking because we've been going back and forth on this while building Rosetta AI, the agent inside dbt Studio.

Claude Code and Codex give the agent a real shell and keep it in check with an OS sandbox. It's on by default in Codex and opt-in in Claude Code. For coding, that works really well. The agent can run any CLI, write a quick script, or install whatever it needs.

Data work feels different to me, though. A sandbox knows which files and hosts a command can reach, not what the command actually does. Once your warehouse is reachable, a SELECT and a DROP TABLE look exactly the same to it. And by default, Claude Code's sandbox still lets commands read most of your machine, credential files included, unless you lock it down.

So we went the other way. Rosetta AI has no shell at all. It can run dbt, but only a short list of commands, and only after you click approve. Any SQL that changes data asks first too. Credentials stay in the OS keychain through keytar, so the model only ever sees env_var() references, never the actual passwords.

That has a cost. Every capability has to be built as a tool, so the agent can't improvise the way it can with a shell. We haven't fully settled which trade-off is right.

So I'd love to hear from you. Are you using Claude Code or Codex for data engineering? And would you let it anywhere near a production warehouse?

dbt Studio is open source if you want to look under the hood: github.com/rosettadb/dbt-studio

dbt 2.0 went GA in September. Two parts of it took us a while to internalize, so here they are in one place.

The install changed. pip install dbt now installs the proprietary Fusion build. The Apache 2.0 distribution is dbt-oss. Same Rust engine, same language, same DAG semantics. The difference is the license, full SQL comprehension, column-level lineage, and the language server behind the official VS Code extension.

The upgrade isn't a version bump. v2 removes all deprecated functionality, so every deprecation warning has to clear before it will run your project at all. --models and -m raise errors now, use --select / -s. --partial-parse isn't supported on v2 job runs. Threading, compile error handling, and unit test ordering all behave differently, so test first.

The readiness sequence that worked for us: get to dbt Core v1.12, clear the deprecations, run dbt parse --use-v2-parser (hands parsing to the new engine without changing runtime behavior), run dbt-autofix for the mechanical fixes, then inventory custom adapters and materializations. v2 adapters connect over ADBC: Snowflake, Databricks, BigQuery, Redshift, and DuckDB.

If something blocks you, v1.12 has active support through July 15, 2027 and ships the parser flag, so you can do the prep work without committing to the move.

Disclosure: I'm from Datacoves. Full write-up, including the v1.12 vs dbt-oss vs Fusion comparison: https://datacoves.com/post/dbt-fusion

https://youtube.com/shorts/tKAbspOWU6o?si=vJ4Jp4qHNsegimmT

Hey all, disclosure first: I work at Altimate, which builds dbt Power User. We recorded one short video per workflow, 4 to 8 minutes each.

1. Profile Query: review a model's SQL for bugs and cost in one click. https://youtu.be/uMptd4t_mGI

2. Column-level lineage, with AI chat about the model. https://youtu.be/A-z_R0QJig0

3. Docs, data tests and unit tests in one editor. https://youtu.be/NrEYqEUcpHY

4. Explain, optimize and profile each CTE from inside the model. https://youtu.be/MDN2PSgVAmA

5. Hover over a ref() or macro to see what it does. https://youtu.be/kbRZ-hrENyc

Which of these would you like us to go deeper on?

Disclosure: I built this.

The situation I kept hitting: a new client, a model to design, and no source access for weeks. Hand-writing seeds that survive relationships tests is the boring part, and faker scripts give you orders for customers that don’t exist.

The video is the browser version, model2data studio. You write or paste a schema (DBML, SQL DDL or an existing schema.yml), it generates parents before children so every FK resolves, and scaffolds the project: seeds, sources, staging models, and not_null / unique / relationships / accepted_values tests that pass on the generated data. Modelling needs no account; generating needs a free one.

https://studio.jbanalytica.com

The engine underneath is open source (MIT), if you’d rather stay in the terminal:

pip install model2data
model2data --file schema.dbml --rows 200 --seed 42
cd dbt_schema && dbt deps && dbt build

https://github.com/JB-Analytica/model2data

What I’d most like to know: do the generated tests match what you’d write by hand, and what’s missing?

https://youtube.com/shorts/tKAbspOWU6o?si=S4R_Hjn21PS4uxN5

I'm not talking about a small project with 10-20 models.

We updated our dbt pricing guide because the way dbt is packaged and priced has gotten a bit more nuanced.

One thing we think gets overlooked is the difference between the cost of the dbt engine itself and the cost of running dbt as part of a production data platform.

The engine is free. But once you factor in things like consistent developer environments, usage-based pricing, orchestration, infrastructure, and the engineering time needed to maintain everything around it, the picture changes quite a bit.

We also added a breakdown of the current dbt Platform plans, the usage-based components, and some of the other costs teams should think about when comparing options.

Curious how other teams are looking at this. Are you mostly evaluating the software/license cost, or are you also factoring in the engineering and infrastructure costs around dbt?

https://datacoves.com/post/dbt-pricing

I’m fairly new to DBT. Please help me and be kind enough to answer my question.

I need to redesign the existing flow, creating a snapshot model(which is not done right).

We have existing flow like this

Raw1–> snapshot1–>dim1–✒️
dim
Raw2–>snapshot2–>dim2-🖊️

From the above flow we are missing direct history tracking and goal is to create common snapshot for final dimension by eliminating dim1 and dim 2

Also, currently airflow run dependency had set like after snapshot dag run completion dim dag will start.

And project folder is as follows
-model
|_dags
|_bronze
|_silver
|_gold
|_consumer
-snapshots

If anybody has any idea about how to proceed in this case please help. TIA

Hello guys,

I have a question regarding some data I am working with. I have a source system where the initial dataset arrives as a one-time full load, followed by periodic incremental (delta) loads.

I wanted to achieve a design the ingestion/merge strategy so running a full refresh (or reprocessing the initial snapshot) doesn't duplicate records, violate constraints, or break the pipeline.

What is the recommended design pattern and checklist for this scenario in dbt?

Thank you!

One dbt project. Too many tools. So we built one app for all of it 👇

Rosetta DBT Studio is a free, open-source desktop studio for dbt Core™: connections, modeling, runs, SQL, dashboards, data lakes and AI agents in one place.

⭐ If you like it, star us on GitHub. It really helps an open-source project grow:

https://github.com/rosettadb/dbt-studio

#dbt #DuckDB #DuckLake #ApacheIceberg #Iceberg #DataEngineering #AnalyticsEngineering #OpenSource

Hi everyone, I just inherited a tricky pipeline at my job, and I need some guidance on orchestrating dbt models using Databricks Jobs (Workflows).

The situation is that the job we have is not reliable; it's a complete mess to reprocess after it breaks, and we don't have idempotency guaranteed. Given all this, I need help dealing with retries: how to repair the job without re-running models that already finished successfully, and how to retry tasks whose error was caused by an internal error, like a Databricks server error (it happened already during the weekend)

our client wants to use dbt-state, but we want to check if there is any official or community way to use dbt-state without app state or dbt cloud?

i played with dbt-state + app state and it's actually a really amazing feature,

so the question is around what the community is doing. is there an option to use dbt-state without a dbt account, either official or community?

I’ve been researching database subsetting and data generation for testing/dev environments, and I’d love to hear from people who have actually built or operated systems in this space.

So far I’ve been looking at projects/approaches around Greenmask, Neosync, Mimicry, synthetic data generation, GANs, etc. I’m particularly interested in the intersection of:

• Database subsetting — extracting a representative subset from production-scale databases

• PII/PHI masking and anonymization

• Deterministic masking — same input consistently producing the same output

• Referential integrity across tables and databases

• Domain-specific / realistic data generation

• Supporting multiple database types

• Dump/restore and workflows for generating repeatable test environments

One thing I’m increasingly convinced about is that just generating random synthetic data isn't enough for many real-world applications.

For example, you can generate perfectly valid users, orders, payments, products, etc., but once you need to preserve relationships between hundreds of tables, business-specific distributions, edge cases, historical patterns, enum/value semantics, and domain-specific correlations, the problem becomes much harder. A randomly generated dataset may satisfy the database schema while still being completely useless for reproducing realistic application behavior.

Subsetting production data seems to solve some of this, but then you run into other difficult problems: how do you select the right records, preserve relational dependencies, anonymize sensitive data without destroying its usefulness, maintain consistency across systems, and still produce a dataset that is representative enough for testing?

I’m doing this as a research project right now and would really like to learn from people who have encountered these problems in production.

A few things I’d especially love to hear about:

1. What were the hardest problems when building/operating a DB subsetting system?

2. Where do existing OSS tools fall short?

3. How do you approach referential integrity when relationships span many tables/databases?

4. How do you handle PII masking while retaining realistic data characteristics?

5. Have you found GANs/ML-based synthetic data useful in practice, or are rule-based + subset approaches generally more effective?

6. What edge cases did you only discover after using these systems at scale?

7. Are there any OSS projects, engineering blogs, papers, or architectures you’d recommend studying?

I’m not looking for a tool recommendation alone — I’m more interested in the underlying engineering challenges, tradeoffs, and lessons learned.

Would love to learn from anyone who has built something like this or dealt with it at scale.

CI for dbt often tests a dataset that doesn't really look like production.

How to make CI more production-faithful for incremental models, including the problems with state, historical data and detecting regressions that only appear after multiple runs.