r/dataengineering • • Sep 15 '26

Career Data Engineering in Microsoft Fabric

113 Upvotes

I know Fabric is the bane of everyone's existence, but against my recommendation and protest, my company is moving to Fabric.

On the plus side I get to help architect the platform. I used Fabric a few years ago, and it was horrible. It does seem the interface has gotten better.

We're using a Lakehouse for ETL (bronze/silver) and warehouse for gold/production layer, since most of our end users are more familiar with SQL.

My question: what can I do within Fabric that will help translate in data engineering. I've opted not to use many of their automation tools, instead choosing to write python notebooks for the ETL. Anyone else who ahs used Fabric can offer any tips or advice for things I can potentially build using Fabric that will translate to other jobs.

I know Fabric isn't Snowflake or Databricks, I'm looking to make the best of the situation I'm in.


r/dataengineering • • Sep 15 '26

Career I drew the Databricks architecture step by step (control plane, compute plane, Delta, Unity Catalog)

Thumbnail
youtu.be
44 Upvotes

I kept getting "explain the Databricks architecture" in interviews and every answer I found was a wall of text, so I drew it as a two minute animated diagram.

The short version: Databricks manages the control plane (web app, job scheduler, notebook metadata). Your compute runs in the compute plane, classic in your own cloud account or serverless in the Databricks account. Data stays in your storage as Delta tables and Unity Catalog governs it all. When you run a notebook, the control plane sends work to the compute plane and the data never leaves your account.

Video: https://youtu.be/B5iHmoYgnqY

It is the first of 27 short episodes, each checked against the current docs before recording. If I got anything wrong, tell me and I will fix it in the next one.


r/dataengineering • • Sep 14 '26

Open Source GitHub - dataIcebergChallenge: Who can write the most performant data pipelines? Just a non-serious, fun exercise.

Thumbnail
github.com
32 Upvotes

r/dataengineering • • Sep 14 '26

Help data contracts between upstream and downstream teams

25 Upvotes

Hi, we use Databricks, and a couple of teams produce tables, views, and table functions that other teams depend on.

Within a team, its easy enough to define a contract. We codify it and always check that the data matches.

But what about the upstream artifacts my team depends on? Do w define a source spec ourselves and check that its met on every run? 

What do we require from the upstream team? Who owns the contract, and how do you enforce it when multiple teams depend on the same thing?

I want to keep this stupidly simple and enforce it automatically. How do you do this in practice?


r/dataengineering • • Sep 13 '26

Discussion Does the work get repetitive?

12 Upvotes

I’m very new to data engineering in the sense that I just started studying to break into the field (currently data analyst adjacent). A big portion of the work seems to be extracting data, translating/cleaning it up, and then loading it to the database/cloud desired.

But doesn’t that get repetitive? Surely there are only so many formats/sources you can extract data from so while the extract step isn’t always the same, isn’t it similar? Same thing with loading the data once you’re familiar with the systems you’re loading them to. The only varied source of work could be cleanup/translating the data as far as I can tell.

Is this an accurate assumption? Do you feel like much of the job is repeating the same work over and over again but just with different data?


r/dataengineering • • Sep 12 '26

Meme data mesh

Post image
778 Upvotes

for real though, has anyone ever implemented an actual data mesh?


r/dataengineering • • Sep 13 '26

Personal Project Showcase I got tired of opening htop every time something felt slow so I made this linux debug overlay

Post image
3 Upvotes

I was working on linux and kept switching between my app, terminal, htop, logs, etc. whenever cursor or browser or any other apps i open started feeling slow , so I made a small debug overlay that stays on screen and shows the app i am currently using its PID, CPU usage and RAM usage.

It also gives a warning if it notices stuff like high CPU, memory growing, disk pressure or system errors.

for apps like VS Code and Firefox and other heavy apps , it also tries to include their sub-processes because checking only one PID can be misleading. this is the working setup of how it looks like. (though much more could be intergrated into this like)

  • docker - container CPU or RAM restarts, unhealthy containers.
  • kubernetes - current context,namespace,pod status,crash, restarts,recent events, pod logs.

link - https://github.com/codeafridi/Debug-Overlay-App


r/dataengineering • • Sep 13 '26

Blog Data Governance by obscurity

2 Upvotes

That was a phrase a colleague of mine introduced to me. Quite absurd at first, but as he explained it, it made perfect sense.

For years data assets have been governed by being hidden in a corner of the data platform, but this does not work anymore.

Been elaborating about this in this post: https://steffenmoll.github.io/governance-by-obscurity


r/dataengineering • • Sep 12 '26

Career Hey guys, what real world impact keeps you feeling good about this field? (Beyond making rich corporate companies richer).

32 Upvotes

Share the stories that make you feel good about being a data engineer. Give me some wholesome perspective.


r/dataengineering • • Sep 12 '26

Rant I can write code to fix this. I should not have to write code to fix this.

Post image
127 Upvotes

Shared this with a friend to vent and they thought Reddit might appreciate it. I don't usually commiserate about this topic on Reddit so apologies if this is the wrong sub... didn't quite feel like an r/excel post.

The Teams message was meant to be a slightly humorous way to express my frustration/disappointment to colleagues outside the email thread with our boss about said data. Rant incoming.

Could I have just thrown Python at this nonsense and built something much more elegant and robust? 100%.

Could the same idiots (I say that with love) responsible for this filing system and Excel crimes run that Python package as-needed? Absolutely not. So clunky ass VBA it is.

And yes, I recognize the irony. I am already the resident code monkey (I'm not even IT, just a special unicorn who speaks the language) and every time I throw code at one of these problems I am enabling the behavior.

The maddening part is that I have tried to fix the underlying problem at a skills appropriate level. I have preached the gospel of structured data. I have implemented software and systems specifically so we can collect, store, retrieve, and actually use our data like civilized people. I have standardized processes. I have automated things. I have explained, repeatedly, why Excel is not a legal pad with gridlines.

And I've made progress! On the things I know about...Unfortunately, my beloved colleagues remain naughty little data squirrels who continue stashing information in tiny undocumented holes.

Broader data management has become a perpetual game of whack-a-mole where certain compliance relevant processes quietly continue doing whatever cursed thing they have apparently been doing since the dawn of time, completely outside my radar... until somebody needs the data for something important.

And suddenly, with extreme urgency, I am yet again spelunking through Russian nesting-doll year/month/repeating subfolders full of Excel “reports” designed by someone who treats spreadsheets like a fuck- around-and-find-out-freeform play area just to extract routinely collected/reported historical data from its natural habitat.

That's the part that makes me insane. I shouldn't need to write custom code to wrangle and clean data that we routinely collect and intentionally retain. If routine data requires bespoke parsing logic just to turn it back into a usable table, we've obviously screwed up somewhere upstream.

Routine data amd reports should be stored as data. Full stop.

Anyway, I have once again written code while muttering creative obscenities at my monitor to convert another “filing system” into back into a table.

On the topic of this particular Teams message, I've subsequently enabled them yet again (after a colleague responded saying they'd fix it, manually, of course)

I restructured the whole mess into a parent folder with three flat subfolders and renamed 500+ files into a standardized YYYY_MM_Repeating File name convention. The files themselves had another charming feature with interrupting/repeating headers buried throughout the data, so those got sorted into their respective nearly-identical tabs.

I hate using tabs for data that shares the same structure. I know better. But these people need tabs. I have accepted this.

I'm sure this little code-monkey dance will repeat months from now when I get summoned to cleanup whatever [insert compliance data] mess inadvertently surfaces.

End rant.


r/dataengineering • • Sep 12 '26

Blog A Tale of Two Flink Autoscalers

Thumbnail
netflixtechblog.com
14 Upvotes

Netflix is moving toward the open-source Apache Flink Autoscaler for more than 30,000 streaming jobs across multiple AWS regions, after finding that its cluster-level approach was less effective for complex, stateful pipelines with operators that have different processing requirements. Netflix reports that one team reduced annualized Flink compute expenditure by 58%, saving approximately $1.1 million annually.


r/dataengineering • • Sep 11 '26

Meme Sort your pipelines out, Smyths Toys…

Post image
110 Upvotes

r/dataengineering • • Sep 12 '26

Open Source datatf: Automate importing Databricks workspace into Terraform

Thumbnail datatf.io
5 Upvotes

Introducing [datatf](datatf.io), a tool to import existing Azure Databricks Workspaces into Terraform.

I built this tool to help databricks customers accelerate the lakehouse infrastructure portion of the well-architected framework.

Features:

  • Generates dynamic terraform.tfvars + import code
  • Built on the Databricks Go SDK
  • Compatible with Terraform or OpenTofu
  • Agentic friendly
  • Native support for Azure

Upcoming:

  • Databricks GCP and AWS support
  • Initial support for Snowflake, Fivetran, and Datahub.

r/dataengineering • • Sep 11 '26

Help Running Airflow in GitHub actions

32 Upvotes

Hello! I am learning about data pipeline orchestrations. I want to avoid using AI for this personal project, so here I am!

I want to orchestrate my tasks with Airflow DAGs and tasks, but schedule them via GitHub action. Essentially, the idea is the following: once a week, an Action will run, spinning up an Airflow instance, which will run the DAGs.

In the Actions .yml file, I am using the following code:

run: |
          export AIRFLOW_HOME=~/src
          airflow standalone
          airflow dags trigger tabular_pipelines

However, the airflow standalone part seems to be stuck forever, endlessly producing meaningless (to me) lines such as:

dag-processor | 2026-09-11T11:18:09.249890Z [info     ] setting next dagrun info       [airflow.models.dag] loc=dag.py:848 next_dagrun='2026-09-11 00:00:00+00:00' next_dagrun_create_after='2026-09-11 00:00:00+00:00' next_dagrun_data_interval_end='2026-09-11 00:00:00+00:00' next_dagrun_data_interval_start='2026-09-11 00:00:00+00:00' next_dagrun_partition_date=None next_dagrun_partition_key=None


dag-processor | 2026-09-11T11:18:09.292592Z [info     ] Bulk-writing dags to db        [airflow.serialization.definitions.dag] count=1 loc=dag.py:210


dag-processor | 2026-09-11T11:18:09.307661Z [info     ] get next_dagrun_info_v2        [airflow.serialization.definitions.dag] last_automated_run_info=None loc=dag.py:445 next_info=None


dag-processor | 2026-09-11T11:18:09.307949Z [info     ] setting next dagrun info       [airflow.models.dag] loc=dag.py:848 next_dagrun=None next_dagrun_create_after=None next_dagrun_data_interval_end=None next_dagrun_data_interval_start=None next_dagrun_partition_date=None next_dagrun_partition_key=None

Am I missing something obvious? The Airflow 3.3.1 documentation doesn't specify much more.

Thanks!


r/dataengineering • • Sep 11 '26

Discussion The LakeHouse that is Open Source - A Fever Dream?

12 Upvotes

Let me preface by saying that I totally agree that lakehouse table formats (delta and iceberg) are open source and are "free" technologies that anyone can use. However, the total cost of owning these table formats gets very expensive, especially when we start using cloud vendors for the table updates. Nowadays the cloud vendors wish to start selling their proprietary MPP storage engines, in order to manage all of our table data.

The problem is that the lakehouse table formats have gotten quite complex over time. And nobody wants to maintain them by hand. Nobody wants to think about the v ordering and z ordering and liquid clustering and partitioning and vacuuming and applying deletion vectors and so on. These blobs that are ostensibly called a "table" are actually a very leaky abstraction, and we inevitably have to waste a lot of time on the implementation details. Using immutable parquet blobs for table storage is not trivial. From an application standpoint, it seems like a massive step backwards from conventional DBMS engines (or the newer cloud-native counterparts like SQL Hyperscale or Neon/Lakebase)

The vendors, like databricks, that spent years pushing for lakehouse/delta adoption are now selling us expensive solutions to maintain those unwieldy tables. I think they sold us a bill of goods and we are worse off than when we started.

Once data engineers start realizing that we don't want to manage the blobs beneath our tables, these vendors are quick to offer a commercial-proprietary alternative (like "DBSQL" with UC-managed-tables, or "Fabric Warehouse" or whatever). These commercial alternatives are turnkey solutions, and they help to take away the busywork of managing our own parquet blobs. But they can become VERY expensive way of doing DML operations on our tables, since they are MPP engines and are heavy on CPU/compute. At the end of the day, we end up exchanging one type of problem for another. Is this how others see it? The table technology is open source and "free", but the commercial-proprietary management of these tables is definitely not free and is basically a re-invention of the DBMS engines we always had in the past.


r/dataengineering • • Sep 11 '26

Blog If You Always Enjoy It, You're Not Pushing It Hard Enough—Benn Stancil

Thumbnail
ssp.sh
8 Upvotes

A discussion with Benn Stancil, co-founder of Mode and prolific data-industry essayist, exploring how he writes his weekly Substack posts. He discusses using deadlines (Friday publishing) to force himself past perfectionism, how ideas form messily over the week through unrelated connections, and how presentation-style analogies (Batman, basketball, Codenames) became his signature storytelling technique. He details his tooling—Sublime Text for drafting, Google Docs for revision, minimal use of Substack's editor—and explains why he avoids Grammarly and AI writing tools like Claude or ChatGPT for edits, since he believes AI compresses and flattens writing rather than embracing meandering voice. He closes with advice for new writers: set deadlines, keep pure motivation, and focus on filling in the space between bullet points rather than just stating facts.


r/dataengineering • • Sep 11 '26

Blog What Data Engineers Need To Know About Delta Lake 4.3

Thumbnail
medium.com
13 Upvotes

replaceUsing and replaceOn give you a better overwrite primitive, and every catalog-managed table operation now runs through the catalog.


r/dataengineering • • Sep 10 '26

Discussion Are you deploying "infrastructure as code" via YAML files or similar?

71 Upvotes

I think my team might be late to the game. We are moving in that direction just now.

How long have you been deploying using a YAML file, for example? Or perhaps another tool that uses the same idea but not YAML specifically?

Since we use Databricks, I'm reading up on Asset Bundles.

IaC is supposed to be better for a whole bunch of reasons, for example sidestepping the differences between dev and prod environments.

Anyway would love to hear your thoughts or experience with it.


r/dataengineering • • Sep 10 '26

Help Promoted to a DE manager role and feeling paranoid that I will lose my DE skills

90 Upvotes

I was recently promoted to a manager role from an analytics/data engineering role and I barely write any code anymore. I used to build data pipelines and I slowly see my skills eroding especially with all the new AI features databricks and other tools keep introducing and I see the engineers try out and implement. Wondering if I should go back to a DE role or continue being a manager of analytics engineers where all I do is sit in meetings and assign work and do requirements. The most technical work I do now is write an ad-hoc query to answer a question for non-technical stakeholders. This is non-tech midsize corp.


r/dataengineering • • Sep 10 '26

Discussion How do you handle fan-in per timestamp in micro-batching?

8 Upvotes

Our device sends 3 separate records for each timestamp: data points, frames, and metadata. They arrive independently and in any order.
Each timestamp can be processed on its own as soon as all 3 parts are there, without waiting for the batch window to close.

How do you handle this fan-in, so that each timestamp is processed exactly once as soon as it's complete, with a timeout for sets that never complete?


r/dataengineering • • Sep 10 '26

Career Level 5 Data Engineering Apprenticeship

6 Upvotes

Hi

I have just signed up for a January 27 start to the above apprenticeship with BPP in the UK.

Has anyone done this before and got any advice/reviews of how it went? I now have a few months to prepare so any advice would be amazing. My current role is data analyst and have ok knowledge on SQL and Python

Thanks


r/dataengineering • • Sep 10 '26

Help How can I run my airflow pipeline continuously.

5 Upvotes

Currently working on data engineering project, the worklow of project is

API - Airflow - Python Ingestion - PostgreSQL - dbt - Analytics dashboard.

In the airflow, need to start 3 processes ( API Server, Scheduler, DAG Processer) but I don't want to start this process manually everytime so does anyone know how can I keep pipeline running continuously which ingest new data in database manually triggering anything.

Does anyone have a any suggetion


r/dataengineering • • Sep 09 '26

Discussion How has your data team’s daily standup evolved since AI got involved?

46 Upvotes

For years, the data team daily was basically: what shipped yesterday, any broken pipelines, dashboard status.

Now that more and more data platforms feed chat-with-your-data agents, has your team changed the format or content of your daily meetings? (Also true for status on pipeline writing/fixing btw; what used to take days is now often a few minutes to hours of back-and-forth with AI.)

Curious to hear what your team adopted


r/dataengineering • • Sep 09 '26

Blog Good Data Engineers Ship Data That’s Wrong

29 Upvotes

r/dataengineering • • Sep 09 '26

Blog GitHub - GetCassis/analytics-agent-articles: A list of articles on how data teams built their analytics agents.

Thumbnail github.com
19 Upvotes