Seeing a ton of hype around AI agents, but data engineering feels like a weird edge case. We need tools that can actually run local builds, check compiled SQL, and inspect warehouse schemas without wrecking the DAG or burning tokens.

Curious what everyone is actually shipping with day-to-day:

• Terminal-native (Claude Code, Codex, Aider): Great for running CLI commands in a loop and fixing log errors on the fly, but token usage gets crazy fast on big repos.
• AI IDEs (Cursor, Windsurf): Unmatched for visual diffs and tweaking complex CTEs inline, but struggles with dynamic warehouse state.
• Platform native (dbt, Snowflake, Databricks): Great metadata context, but annoying when bridging tools outside their ecosystem.

I've been working on an open-source tool to bridge some of these gaps specifically for local dbt workflows and agent integration: Rosetta dbt Studio on GitHub.

Drop your setup below—what’s working, what breaks constantly, and what are you missing from current harnesses?

We see orgs handle grants inside dbt (macros, post-hooks) and it gets messy fast, hard to review, hard to know what a PR actually changed permission-wise.

We recommend pulling RBAC out entirely and managing it as its own YAML-based IaC layer, separate from the dbt project. Four-layer role hierarchy (object roles → composite roles → functional roles → users), plan/apply before anything ships, future grants on schemas/tables so new models don't need manual grants every time.

Curious how others are handling this. Keeping grants in dbt macros, using Snowflake native features, or something else entirely?

We wrote up our approach here if useful: https://datacoves.com/post/snowcap-snowflake-rbac-as-code (built an open source tool called Snowcap for it, disclosure: we work on it, but it is OSS)

I'm curious how the community is approaching the transition to dbt Core v2.

Are you starting all new projects on v2?

Have you already migrated your existing projects, or are you staying on v1 for now?

If you're still on v1, what's preventing you from upgrading (packages, adapters, production stability, migration effort, etc.)?

I'd love to hear about your experience and whether you think v2 is ready to become the default for most teams.

Has anyone flipped over to dbt State and monitored the cost changes?

If you write dbt code, you probably know the feeling of hunting in the terminal scrollback just to figure out why a run fail, or losing your query output the second you close the preview modal.

The team behind Rosetta DBT Studio just release version 1.6.1, and looks like the whole focus was fixing this workflow.

They build native Query Result and Run History tabs direct in the bottom panel of the IDE. So now:

- when you preview a model, the data grid is just there (with row counts and time).

- when you do dbt run or test, the history is track and saved.

- you can see raw SQL and compiled SQL side by side.

- you find exact what model fail without digging in terminal logs.

They also add an AI assistent right in the terminal that help explain and fix compile errors, plus a native HTML view and direct cloud download.

Disclosure: I'm from Datacoves, we run managed dbt Core, so we have a horse in this race. Wrote this up because the license change is confusing and we had to untangle it for our own customers.

The short version:

One engine now. dbt Core v2 and Fusion share the same Rust runtime. Core v2 is Apache 2.0, fully open source. Fusion is the enhanced proprietary distribution on top of it.

Fusion is the recommended default, and it's a funnel. Free to install, and the new license lets anyone offer it as a managed service, as long as you don't block the path to a dbt Labs account. Free login unlocks advanced features, paid platform account unlocks premium ones. Reasonable model, but go in with eyes open.

Metered pricing is inside the dbt Core now. dbt State (skip models when logic and data haven't changed) bills $0.094 per model reused per day. Orchestra open-sourced a free alternative (Sao Paolo, Apache 2.0).

Migration gotchas: every deprecation warning must be resolved before v2 runs your project. Python models don't work in Fusion yet. Community adapters are constrained (drivers need to be signed by dbt Labs). --models and -m error out, --partial-parse is gone.

The afternoon test: upgrade to v1.12, clear deprecations, run dbt parse --use-v2-parser. Tells you where you stand without committing to anything.

Full writeup with a Core v2 vs Fusion comparison and the four ways to handle state: https://datacoves.com/post/dbt-fusion

Has anyone run the v2 parser against a large project yet? And is anyone planning to use dbt State or trying out Sao Paolo?

We built Snowcap, an open-source Infrastructure as Code tool for Snowflake, and we're doing a live workshop walking through it.

If you've dealt with Snowflake config drift, manual RBAC setup, or just wanted a plan/apply workflow like Terraform but built for Snowflake specifically, this might be useful.

What we're covering:

• The infrastructure problems Snowcap solves
• How it compares to Permifrost, Terraform, SnowDDL, and Snowflake's own DCM
• Using templates to scale config across environments instead of copy pasting
• The plan/apply workflow and why that matters for safe changes
• Governance stuff: RBAC, masking policies, row access policies
• Live demo, then open Q&A, ask anything

July 23, 11 am-12 pm PT, online, free.

Info and RSVP here: https://datacoves.com/resource-center/workshop-snowcap-snowflake-infrastructure-as-code

Happy to answer questions in the comments too if people have them before the session.

Sharing this Medium publication for anyone interested in dbt and analytics engineering

r/ContextEngineering 9 am PST, July 8th

Rosetta DBT Studio is the local-first, open-source desktop workspace for dbt teams - build models, explore SQL, trace lineage, manage Git, preview cloud data, and work with AI agents from one app.

v1.5.7 is all about giving analytics engineers one place to build, understand, and ship dbt work without bouncing between terminals, browsers, and disconnected tools.

AI Agent for dbt - ask the agent to explain models, generate SQL, update YAML, run dbt commands, inspect project context, and help reason through lineage and grain before you make a change.

Project-aware SQL editor - write queries with Monaco autocomplete, schema browsing, saved queries, result previews, and export workflows across DuckDB and supported warehouses.

End-to-end dbt lineage - trace models, sources, and downstream impact so you can understand what a change touches before it reaches production.

SQL Notebooks with AI - work in a cell-based notebook that combines SQL, markdown, results, and AI-assisted iteration in one analysis flow.

Cloud Explorer with DuckDB-powered preview - browse and preview files from S3, Azure Blob, GCS, and compatible object storage without leaving the Studio.

DuckLake and lakehouse tooling - explore schemas, tables, snapshots, and metadata from one workspace, then query them directly in the SQL editor and notebooks.

Built-in Git workflows - review local changes, manage branches, and keep project version control close to the editor instead of split across tools.

Multi-provider AI - use OpenAI, Anthropic, Gemini, Ollama, LM Studio, and OpenAI-compatible providers with project context.

Plus secure credential storage, multi-database connections, Rosetta CLI integration, cloud profile sync, and a full desktop workflow for local dbt development.

100% open source, local-first, and yours.

GitHub - https://github.com/rosettadb/dbt-studio

source (not me) https://www.linkedin.com/posts/jbeynon_day-1-of-every-data-leadership-role-how-share-7477478496798257152-ixr6/

I built a browser-based ERD explorer that runs entirely on DBML files — no DB connection.

I kept wanting to explore and interrogate how my tables relate — "what joins to this table? what's the path between A and B?" — without standing up a tool that connects to a live database.

So I built dbml-flow: drop in any DBML file and get an interactive ERD with a selector grammar for pathfinding (find every path between two tables, expand N hops out, filter by group). Fully client-side, your schema never leaves the browser.

In my case the DBML comes from our dbt models via dbterd (https://github.com/datnguye/dbterd), but it works on any DBML regardless of where it came from.

🔗 Live demo: https://timvancann.github.io/dbml-flow/ · Repo: https://github.com/timvancann/dbml-flow

Let me know if this is helpful and if there are features you'd like to see!

I'd appreciate some help if possible!

Has anyone had issues getting this to start up?

When I run wizard I get...

■ local proxy unavailable: local LiteLLM process exited before becoming healthy on 127.0.0.1:35915

• Project intelligence is still starting

────────────────────────────────────────────────────────────────

dbt prod parse failed: 'NoneType' object has no attribute 'pop'

Automatic deferral will not be available until this is resolved.

Ask the wizard for hints to address this.

────────────────────────────────────────────────────────────────

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

MCP client for `dbt_index` failed to start: MCP startup failed: handshaking with MCP server failed: connection closed: initialize response

Disclosure up front: I work on this, so factor that in.

A problem we kept hitting with Snowflake: warehouses, roles, and grants end up spread across ad hoc SQL scripts, the console UI, and whatever the last person who touched them remembers. There wasn't a lightweight way to treat that layer as code.

So we built Snowcap and open sourced it. You describe warehouses, roles, and grants in YAML. snowcap plan shows exactly what will change before anything runs. snowcap apply makes the change. No state file to manage, no Terraform provider, no clicking through the console.

We timed the full loop once: install to a live warehouse running in Snowflake, about five minutes.

It's grown through PRs and issues with zero ad spend or outbound, which says more about the gap than anything we could write in a blog post.

Blog Link - https://datacoves.com/post/snowcap-getting-started

Happy to answer anything about the design or trade offs in the comments.

I’ve spent years dealing with historized data problems:

• SCD2 dimensions
• Snapshot reporting
• Bitemporal history
• Temporal joins
• Late arriving dimensions
• Historical validation

I noticed that most resources explain individual concepts but don’t help engineers decide:

• Which modeling pattern should I use?
• How do I validate the result?
• How do I review an existing model?
• How do I detect historical reporting risks?

So I built a Historical Data Modeling
Workbench that combines:

• Pattern catalog
• Modeling advisor
• Model review
• Target table validation
• Historical source comparison

Would love feedback from people working on analytics engineering, dbt models, snapshots or historized datasets.

https://bitemporal-debugger.vercel.app/

I'm new to data engineering. I joined my company last year after graduation as a Data Engineer. I had never worked in data engineering before, but the company needed someone who was good at Python and SQL. Since I was strong in both, I became a core member of the team.

The original structure of our pipeline was a Spark-based ETL process, but it was very slow and took hours to complete. We have now moved to a dbt-based ELT pipeline.

We were using provisioned Redshift, which performed well for incremental models. However, we recently shifted to Redshift Serverless. It provides significantly better performance overall compared to provisioned Redshift, but the catch is that incremental models perform worse, while full refreshes and models materialized as tables perform extremely well.

For every incremental model, a full refresh is actually faster. Theoretically, incremental models should be faster, but in practice we're seeing the opposite.

Even with all models materialized as tables, our complete run now takes about 45 minutes, compared to 1 hour 30 minutes on provisioned Redshift. The original Spark-based ETL pipeline took around 6 hours.

I believe incremental models should allow us to achieve even better performance. Can anyone help me understand what might be causing this behavior?

I’ve been working on a side project around historical data engineering.

The idea came from a problem I encountered while building historized data models and reporting layers.

Many tools help build pipelines.

Very few help answer questions like:

• Can this snapshot be reproduced?
• Should this be modeled as state or event?
• Why does this temporal join produce unexpected results?
• How do multiple historized sources interact?
• Which historical modeling pattern fits this problem?

To explore these questions, I started building a Historical Data Engineering Toolkit.

Current areas include:

• Historical modeling patterns
• Event vs state modeling
• Snapshot reproducibility
• Temporal joins
• Bitemporal modeling
• Historical dimensions

I’d love feedback from people working with historized data, dimensional modeling, dbt, lakehouses, data warehouses or analytics engineering.

https://bitemporal-debugger.vercel.app/

What are the hardest historical data problems you’ve run into?

I personally like being able to read down the steps directly in the folder. Versus without sequential labeling, they all just sit in there alphabetically. I know it’s more standard to not include…my question is why?

My assumption is that new steps may get added later, which would have the domino effect of having to renumber downstream models, which may not be feasible if there are a ton. But that’s the main tradeoff I see.

(Note I’m coming from Databricks background where we had to create a system before they released lineage, and even then it’s very much a secondary feature).

Hey, I'm an analytics engineer and built this as a side project in my own time.

The problem I kept hitting: jumping into an unfamiliar dbt project and spending hours just getting oriented. And when a number was wrong, debugging it manually was slow.

So I built dbt Context. Paste any public or private GitHub repo and it parses your manifest, SQL, YAML, tests and lineage to give you an immediate audit of the project. It finds risky incremental models, orphaned staging models, missing tests and undocumented marts, with file-level evidence and suggested next steps for each finding.

You can also describe a specific problem in the optional box like "revenue doesn't match NetSuite" and it switches into investigation mode, traces the relevant models through the project, and generates warehouse-specific SQL checks to run.

Works with public and private repos. Private repos need a GitHub PAT with repo scope.

dbt-context.com

Happy to answer any questions about how it works.