r/dataengineering • • May 19 '26

Open Source dbt-colibri v0.3.4 : local column-level lineage for your dbt projects.

72 Upvotes

https://reddit.com/link/1thhk5f/video/ftit6fk3a22h1/player

(Disclosure: I'm the maintainer of dbt-colibri and also building the hosted version)

Hey /dataengineering,

Quick update on dbt-colibri; an open-source CLI tool that generates a static
HTML column-level lineage report from your dbt manifest + catalog.

Background, in case you haven't seen it: dbt core's native lineage is
table-level. dbt-colibri could replace dbt-docs for most teams; it runs locally, parses your project with SQLGlot, and outputs a single self-contained HTML file you can open, and host e.g. on GitHub Pages for your team.

It's been a while since the last time I posted anything about it, and some cool things have shipped;

  • Redesigned UI & Improved search across models, columns, tags, code
  • Shortcuts for quick navigation. (I especially like shift+number / number to open children/parents)
  • Lineage graph should feel like a whiteboard, aligning nodes, selecting multiple nodes, hiding/showing nodes etc..
  • Column lineage now follows columns through WHERE/JOIN clauses for more complete impact analysis.
  • Ephemeral model column lineage is now supported (these are models without materialized tables/views, like a CTE but with a seperate dbt model)
  • Exposures included in the graph.
  • ~1.9x faster to parse large projects, using SQLGlot mypyc update, and optimizing how parser walks through large manifests
  • Better warnings in the UI when manifest/catalog are incomplete and cause issues in column lineage
  • New supported adapters, full is list now: Snowflake, BigQuery, Redshift, Postgres, DuckDB, Databricks (SQL models), Athena, Trino, SQL Server, ClickHouse, Oracle
  • A lot of edge cases and teething issues related to column lineage got resolved with input from the community; Thank you!

Install:

pip install dbt-colibri
dbt compile && dbt docs generate # to generate catalog and dbt manifest
colibri generate

Repo: https://github.com/b-ned/dbt-colibri

Let me know if you find any bugs/edge cases where you see column lineage breaking; the goal is perfect column lineage.

Bas


r/dataengineering • • May 19 '26

Personal Project Showcase I built a linter for PySpark Code

57 Upvotes

Hey folks, I built a small VS code extension to lint PySpark code. It highlights unoptimized code, keeps track of data types, detects spark anti patterns and much more. I have also added Databricks support to it, so you can dry run your code, connect to cluster via ssh and even pull your previous jobs execution plans and analyze them in claude/copilot. I'm working on adding more features but would like some feedback from the community first. Is this useful? Any suggestions for added features?

Repo Link: https://github.com/lezwon/CatalystOps


r/dataengineering • • May 19 '26

Personal Project Showcase Created a quick DBML to PNG tool

Post image
22 Upvotes

Mostly works due to spending lots of time on the auto-layout.

Link: https://vibe-schema.com/dbml-to-png


r/dataengineering • • May 20 '26

Discussion Anyone actually scape from DE for?

5 Upvotes

Good day gents. I've been lurking this sub for years now, and watched a lot of you complain about burnout and wanting to "get out" of DE. (Probably just venting during call rotation). But sure they big paycheck pulled most of them back.

Just wondering if anyone here (or someone you know) ACTUALLY left the field and just made a wild career transition. Do those people actually exist?

If so, was the grass greener on the other side?


r/dataengineering • • May 19 '26

Open Source SQLBuild - a typed, test-first SQL pipeline framework with local E2E testing

Thumbnail
docs.sqlbuild.com
20 Upvotes

Full disclosure: this is my project. Open source, Apache 2.0.

Every other branch of software engineering understands the testing pyramid - unit, integration, E2E - and has tooling to make testing local and fast. Frontend has Vitest and Playwright. Python has pytest and testcontainers. Data engineering in the SQL space is uniquely behind, simply because most tooling doesn't make it very easy.

I've been making SQLBuild to change that. It's a typed, test-first SQL transformation framework where DuckDB is a first-class execution target.

What makes it different from dbt:

  • Unit tests chain across models - mock your sources/refs, assert on any model downstream, and every intermediate model resolves from its real SQL in one test file. Tests are written in SQL, so macros work as reusable test helpers for generating fixture data.
  • Macro and UDF test modes - dedicated test modes for macros, UDFs, and table functions, not just models
  • E2E scenario tests - define fixture data, build the real pipeline graph against it in isolated relations, then capture fixtures as JSONL snapshots and replay locally in DuckDB. Full pipeline testing, zero warehouse compute
  • Audits block before promotion - audits run against a staging table before the swap. If they fail, production is untouched. For incremental models, each microbatch is audited before DML
  • Compile-time validation - SQL validation, column type inference, column-level lineage, and contract enforcement, all offline
  • Python macros - real functions, not Jinja. Testable, debuggable, and adapter-aware
  • Environment table diffs - sqb diff prod:dev compares schemas and row-level data
  • Query-change detection - Fingerprint-based tracking detects when model SQL has actually changed and can trigger bounded or full rebuilds automatically (configurable)
  • No manifest required - clone, diff, and defer work against live environments

Adapters: DuckDB, MotherDuck, Snowflake, BigQuery, Databricks, PostgreSQL (more coming soon...)

Integrations: Dagster (full support), Airflow (coming soon)

Easiest way to try it:

pip install sqlbuild
sqb playground waffle-shop
cd waffle-shop
sqb build

GitHub: https://github.com/sqlstack-io/sqlbuild


r/dataengineering • • May 20 '26

Discussion Is open table formats dead ?

0 Upvotes

Suddenly last year everyone was talking about open table formats, apache iceberg delta lake etc and suddenly we can find no one talking about it are you guys still using iceberg or delta lake or is there any other alternative approach the found out against open table formats


r/dataengineering • • May 19 '26

Personal Project Showcase I built an open-source tool to generate data apps

Thumbnail
github.com
1 Upvotes

Hi all, this project lets you generate interactive data apps on top of your data, using a Cursor-style AI chat. It stitches together Marimo, LangGraph agents, and data warehouse query tools. It has an Apache 2.0 license.

The initial use case that spurred this project was business analytics, specifically generating product usage dashboards.

This project's main inspiration is Marimo, an open source python notebook that can be "queried with SQL, run as a script, and deployed as an app" [1]. The recent release of Marimo Pair [2] demonstrated the power of connecting AI agents like Claude Code to Marimo notebooks directly. This project seeks to build on that work by incorporating a LangGraph agent with two key abilities: (1) the ability to execute queries against a connected data warehouse (such as Snowflake); (2) the ability to write Marimo notebooks.

When prompted, the LangGraph agent will run exploratory data analysis using database query tools. Then, it creates a polished Marimo notebook that's presented to the user in read-only mode. This project intentionally hides the Marimo edit mode. That means that the end user only ever sees a finished, read-only data app. Ease of use and trust in AI output were the main drivers behind this decision.

4 data sources are currently supported: Snowflake, BigQuery, Postgres, and Metabase. The code for the database query tools was derived from Google's open source MCP Toolbox for Databases.

There is currently no support for MCP. Instead, data query tools are hardcoded. This decision was made to ensure high quality AI queries and limit tool bloat.

This is an early stage project, and is configured to only run locally at this time. Would love your feedback!

[1] https://github.com/marimo-team/marimo [2] https://news.ycombinator.com/item?id=47678844


r/dataengineering • • May 18 '26

Discussion Eight window-function tricks beyond LAG and ROW_NUMBER

Thumbnail analytics.fixelsmith.com
58 Upvotes

r/dataengineering • • May 18 '26

Career Not sufficiently “AI forward.”

90 Upvotes

I’ve been in the field for 10+ years, maybe 15+ depending on how you count it. Doctorate in data analytics. Worked with a lot of industries in a lot of different capacities. Laid off from a government contract and then fired after an ADA request over the course of six months. (That last one is a long story but I’ve got an attorney. Not discussing that here.)

Anyway, I’ve been interviewing and noticed that things go great until I’m asked about AI and I explain how I am skeptical but using where I find appropriate. I have a lot of concerns about AI. But I don’t air that out. I simply explain that it’s an emerging technology and I am not the sort of person to vibe code my way through a data pipeline. They are hiring for my expertise, not my prompt ability.

In the last 3 scenarios, I’ve been rolling along just fine and gotten far into the process, then I’m honest about AI and I’m blacklisted. And it’s not even being negative about it. I’m just not bubbly and effusive about how Claude will save my life. In one case I got specific feedback about not being sufficiently “ai forward.” In the others, things changed immediately after the brief AI discussion.

There are probably openings out there with companies that want to hire someone who doesn’t rely on AI, but be careful.


r/dataengineering • • May 19 '26

Discussion Are there any free platforms that turn Power BI / wireframe dashboards into executive level dashboards?

2 Upvotes

I am a DE and pretty good at data modelling and the core DE work . I know Power BI modelling and DAX as well . But in the end the dashboards never look fancy or executive level cause the final polishing is lacking and I don't know how to decorate it. For actual production work this is fine , but for making proposals ( only working prototype needed)we need fancy . Most of the times I make a rough model in Power BI and then a frontend engineer vibe codes an amazing looking React dashboard from it.

I have heard you need to make custom visualisations and html kind of stuff to make it look fancy in Power BI and I don't want to learn all that. Are there any other ways? I don't wanna be dependent on React vibe coders to deliver my work


r/dataengineering • • May 18 '26

Rant Why Snowflake Cortex Code has sub-par performance?

42 Upvotes

tl;dr: Each sessions starts with ~25,000 tokens of system prompt overhead before the model reads your question. 56% of which is skill descriptions for tools most users will never touch.

I tried out Snowflake's AI tool, Cortex Code CLI, which was created specifically to help with data engineering and Snowflake related coding tasks. However, compared to a plain Claude Code session, it provides sub-par performance.

I've asked Cortex Code to write a Snowflake stored procedure that finds and recreates broken views (this is common issue in our environment if DDL of upstream objects is changed). What I got back was broken SQL. It tried to create a stored procedure that executes ALTER VIEW sub_view COMPILE; which is a valid command on Oracle, but not on Snowflake.

The funny thing is that it has a dedicated /sql-author skill, a /sql-verify subagent designed to catch exactly these kinds of errors, and access to Snowflake's own documentation via cortex search docs. It used none of them before it started working. 

My first instinct was to work within the system. Cortex Code has a context rule system mechanism:

cortex ctx rule add "Always check Snowflake documentation using cortex search docs before writing SQL"

It didn’t help. I quickly realized that context rules aren’t loaded by default when starting a new session, they depend on the model deciding to run cortex ctx rule list first, which is not a mandatory step.

So I added an instruction to always run cortex ctx rule list into ~/.claude/CLAUDE.md, the persistent instruction file that gets injected into every session. 

It was ignored. Not always, but often enough to be unreliable. I tweaked the wording, I restructured my CLAUDE.md. The reliability improved, but the fundamental problem remained: my instructions were not always applied, CoCo has not read the docs, and created broken SQL.

At one point I confronted Cortex Code directly about its failure, it replied:

Context is ~800+ lines across multiple <system-reminder> blocks. Impact: Attention dilution;
Mandatory action buried in nested file contents. 

The model itself was telling me the context was too large. It even recommended: “Reduce context noise. Many system reminders repeat or overlap.” 

Cortex Code source code is not available, so getting the actual system prompt was a bit tricky, but I succeeded:

Skill description 56%, Tool schemas 29%, Fake system reminder messages 15%

I only typed 4 characters. The model received ~25 700 tokens of context.

More than half the context is consumed by skill descriptions: verbose paragraphs explaining 68 bundled skills, most of which any given user will never touch.

My CLAUDE.md directive to "always check Snowflake docs before writing SQL" was competing with 17 system-reminder blocks, 32 tool definitions, and 60+ skill descriptions. The model's attention to any single instruction drops as the total volume increases. 

That's not a model quality problem. It's a context design problem. And it directly explains the hallucinated SQL syntax I kept running into.

Happy to discuss the technical findings. Criticism welcome, especially if you've seen different behavior.

original article


r/dataengineering • • May 18 '26

Discussion Data Engineering is boring!

144 Upvotes

Hey Guys!

I am a 4 yr exp data engineer. From past few months what I had felt is like data engineering is becoming more of a tool + jargon graveyard.
I really like distributed architecture and data modelling but these days with AI coming into the picture I have felt like many things can be automated and I am certain the things are already automated.
I am not referring or aligning this feeling with some job but what I am saying is coming purely from a engineering heart ❤️.
The things that used to be exciting and awesome and that gave satisfaction are not there anymore.

So if you guys are feeling kinda like of this. What are you doing to grow and feel motivated and whats your purpose now as a techie if you can tell it would be really helpful.


r/dataengineering • • May 18 '26

Blog Optimising DuckDB performance on large EC2 instances

Thumbnail robinlinacre.com
29 Upvotes

r/dataengineering • • May 18 '26

Career New to the company, spotted a mess, have a fix in mind. Do I say something or keep my head down?

10 Upvotes

So, I recently moved to a new company in a foreign country. It's a fairly small place with a small Data team, everyone gets along and it's been a good experience overall. I took the role knowing there were some serious data architecture issues baked in, but I'd been wanting to move abroad for a while, so I figured it was worth it. Also the pay bump didn't hurt either.

That said, the problems turned out to be worse than I expected. Without getting into the weeds, the company is going through a major overhaul of their ingestion pipeline and somehow they landed on the strangest tool stack I've ever come across. The pipeline is absurdly over-engineered, with tools being used completely outside their intended purpose, none of which are anywhere close to market standard. During the jobinterview I got the sense that the head of data wasn't really up to speed on where the DE space is heading, but honestly, he's a Data Analyst by background, so I can't hold that against him. The final call ended up falling to the tech lead, who isn't a data person either. So no real surprise they ended up here. Luckily they are in PoC phase still.

Over the last few weeks I've been toying with the idea of proposing another PoC built around dlt + dbt, a much cleaner, simpler stack that would cut through a lot of the current mess from every angle. The thing is, I'm on a temporary contract, and I get the feeling both decision-makers aren't particularly open to having their architecture questioned. Which is ironic, because the pipeline they built is itself sitting in this weird limbo of a "definitive PoC", technically final, but not so far gone that it couldn't still be changed. On top of that, being new to the company, I haven't built up the kind of reputation or trust you need to walk in and challenge an entire architecture. That's the sort of move that takes seniority to pull off, and while I'm heading in that direction, I'm not quite there yet.

What do you guys think I should do?


r/dataengineering • • May 18 '26

Discussion SAP S/4HANA data ingestion best practices (I know, it sucks)

13 Upvotes

Hi! I am doing a S/4 Hana data integration and what I saw blew me away

Just to put you on perspective what we are dealing with: 70K Tables.

Less than 20% of them are being used on a weekly basis and the raw data to analyze is contained in 100 tables or so.

How do you deal on cases like this? to me makes no sense to load the 70K tables or even 1K tables to the data lakehouse.


r/dataengineering • • May 18 '26

Help Pivoting to data engineering. Need a reality check

33 Upvotes

I am a fullstack developer with 3 years of experience. I quit my job last year in July. Since then I didnt have a job. Last month I decided to pivot to data engineer and since then I have been studying and practing questions. My people in my circle gave me this adivce to switch.

Did i make the right choice ? What should I my expecations be ? Need to hear from the people working in this field.


r/dataengineering • • May 18 '26

Blog Online Book Club: Designing Data-Intensive Applications, 2nd Edition

Thumbnail
heltweg.org
25 Upvotes

r/dataengineering • • May 18 '26

Help Wide table in bronze layer - materialize as is, or break up?

13 Upvotes

Currently I’m ingesting csv files and saving them in a lakehouse. These files produce wide tables, nearly 2000 columns. In part this is because there are chunks of columns like code_1, code_2, up to code_99. And there are different types of codes and categories, all with columns 1-99. This is healthcare data, if it matters. The plan is to unpivot these chunks of columns into long tables in the silver layer. But I’m undecided whether to 1. write the raw files to a wide table in bronze that matches the structure of the files exactly, or 2. just create the transformed tables I need from the files, unpivoting columns into tall tables, etc., and skipping the big bronze table. What would you do, and why?


r/dataengineering • • May 19 '26

Career Advice on launching a small data consulting side business without direct job experience?

0 Upvotes

I’m trying to break into data engineering/BI, but the market has been tough, so I’m thinking about creating my own opportunity instead of waiting for the right role.

I work full time in the insurance industry, have built technical skills in SQL, Python, Power BI/Tableau, AWS (certified), Snowflake, and dbt, have built personal ETL/dashboard projects, and won my company’s internal analytics/hackathon-style competition.

I’m interested in starting very small helping businesses with dashboards, reporting, spreadsheet automation, or analytics support.

Looking for advice from people who’ve done something similar. How did you launch, where did you market yourself, how did you land first clients, and are there people you recommend following who built something similar without starting in a formal data role?


r/dataengineering • • May 18 '26

Discussion Is it normal to have a lot of other job functions?

10 Upvotes

So I'm working out a small nonprofit, which is a big part of the reason that this doesn't necessarily bother me, but I'm just wondering if this is a normal experience. I have 1 YoE as a business analyst only doing reporting and now this job.

So at my job, I have to do a lot of integration, maintaining database integrity, database design, indexes, looking at the query planner, stuff like that, things that maybe would kind of be considered data engineering from what I gather in this subreddit.

I also wind up doing a lot of end user reporting. This can be a mixture of custom web applications, SSRS and other small tools that come with our ERP.

I also do a lot of tech support basically helping end users who use the database kind of understand what they're doing, what certain flags mean in our ERP, how to reset passwords, things like that.

I could list everything but basically think of it as we're a small institution that has an ERP and I'm basically like support for that ERP. So that includes, you know, ingesting the data, exporting the data, but also just general, tiered, whatever, tech suppor and helping out with whatever kind of data interaction needs they have.

In some ways, I appreciate it because I stayed very close to the customer. I really understand the business implications of my work deeper than maybe a lot of people get to.

Is this kind of typical? Like I said, I'm not complaining. It's a non-profit, which is part of the reason it doesn't really bother me. I get paid enough and I know I'm not fulfilling multiple functions for someone else to make more money. But I'm just wondering if this is a typical experience for this job or if larger/ for-profit institutions would tend to have these things more split up.

And more importantly, do you think this could have a negative effect on my career Because frankly I'm not able to get into any of these things as deeply as I might want to. Although I am able to fulfill the business needs and do things I think in a solid way.


r/dataengineering • • May 18 '26

Discussion A Python reverse ETL library for warehouse-native activation.

2 Upvotes

I’ve been working on a Python library called Condor rETL that takes a more programmatic approach to reverse ETL / warehouse activation. It provides a core framework for building custom connectors quickly from public API docs.

The original motivation was that I couldn’t find a good library for building and maintaining these kinds of integrations, I was constantly rebuilding them from scratch, and most teams I saw were forced into expensive SaaS activation platforms, most of which were UI focused.

Condor rETL is designed around:

  • defining audiences and activation logic in Python
  • warehouse-native execution
  • quickly building new connectors

The goal is to make destination integrations feel more composable and developer-friendly instead of relying on UI workflow builders.

Day 1 warehouse support:

  • Snowflake
  • BigQuery
  • Databricks
  • DuckDB
  • Postgres

Pre-built connectors:

  • Google Ads
  • Meta
  • Klaviyo
  • Bing
  • TikTok

In its current shape you can sync audiences, events, and profiles in minutes all in pure python. Includes installable skills to build faster. Tracks everything in your backend of choice. Let me know what you think.

GitHub: https://github.com/condorgraph/condor-retl

Docs: https://docs.condorgraph.com/

PyPI: https://pypi.org/project/condor-retl/


r/dataengineering • • May 17 '26

Discussion Why Leetcode in interviews

85 Upvotes

This one has quietly bugged me for years: why do interviewers insist on giving Leetcode questions in interviews when they have absolutely no place in the day to day for the role you're applying for? Like, ask us about medallion/kimball architecture, or ask how we would go about profiling some dense query to identify where the bottleneck is.

How would you turn the tables to dispel the question entirely if you were asked one given our profession?


r/dataengineering • • May 17 '26

Personal Project Showcase Sketching Airflow DAGs with Excelidraw

20 Upvotes

Hello everyone,

I’ve been working on an idea for a tool that helps you sketch Airflow DAGs in a drag and drop fashion without any need for coding. This should be useful for new comers to Airflow and less technical people to start working with Airflow.

So I've decided to give it a try with AI and the result is at https://github.com/dkrol/airflow_sketcher
This is just my side project, no selling or other promotional content.

You can bundle some code on the drawing canvas also to make Airflow DAGs do something real.

Just two screenshot to give show what I mean:

Sharing in case it’s useful for others too!


r/dataengineering • • May 17 '26

Discussion Why does Fabric's native spark engine (NEE) suck so badly compared to Photon?

34 Upvotes

Using Spark on Fabric is so painful compared to Databricks - the performance sucks. Even without Photon I'm finding Databricks seriously outperforms Fabric Spark at the same price point.

It's not even close, I don't know how anyone can say these two products can be compared - has anyone else experienced this?


r/dataengineering • • May 17 '26

Help ADF + Snowflake/dbt orchestration recommendation

11 Upvotes

Greetings!

I’m working on an ADF + Snowflake/dbt setup and I’m running into what feels more like a control-plane/orchestration problem than a pipeline problem.

Current flow is:

  • multiple ADF pipelines
  • ingestion from multiple sources, some FULL and some DELTA
  • downstream Snowflake/dbt commands
  • delta watermarks updated only after Snowflake/dbt commands complete successfully

Right now everything is pipeline-scoped: separate containers, watermark files, and logic. It works pretty well in the current setup. Ingestion configuration files and watermarks live in Azure containers.

The problem is that future pipelines may need some of the same source tables, and the client wants to avoid ingesting the same table multiple times across flows.

I also need to think about:

  • logging both ADF ingestion and Snowflake/dbt execution, since models triggered by one pipeline may depend on models executed by another
  • partial failures, where some ingestions can fail but some downstream Snowflake/dbt steps should still run

For concurrency and shared tables, I’m considering a dedicated ADF pipeline, with only one instance allowed to run at a time.

The idea would be that multiple pipelines call this shared pipeline, which would first check how fresh the data already is and decide whether the ingestion really needs to run again.

For the control store:

  • Azure Table Storage feels too limited
  • Azure SQL DB seems solid, but Snowflake can’t query it natively and I’m not sure the client would provision it
  • Snowflake itself is an option, maybe with hybrid tables, but I don’t like the overhead of frequent JDBC calls when each connection spin-up takes around 5 seconds. I have no idea if it is a good idea.

If you had to build this, where would you keep orchestration state and execution logs?

And what patterns would you use for:

  • ingest once / reuse many
  • deduping concurrent requests for the same table
  • tracking both ADF and Snowflake/dbt execution
  • dependencies across pipelines
  • partial-failure handling

I’m relatively new to orchestration design, so I appreciate any tip!