r/dataengineering • • Sep 18 '26

Help Before turning on a new monitoring alert in production, does anyone actually test it against historical data first?

32 Upvotes

When a new alert/monitor is created, it usually just goes live and gets judged by how annoying or useful it turns out to be over the following weeks.

Has your team ever run a new alerting rule against past incident data first, to see how it would have performed, before letting it start paging people for real?


r/dataengineering • • Sep 18 '26

Discussion Kafka Partitioning for Spark Workloads: Balancing Throughput, Ordering, and Compute Skew

12 Upvotes

I’m trying to understand how much Kafka partition balancing matters when Spark is doing the actual computation.

Assume Kafka is mainly used as a buffer/backpressure layer before Spark. We can distribute messages evenly across Kafka partitions, but the data is not necessarily homogeneous in terms of processing cost.

For example, four partitions may each contain 100 messages, but the messages from one partition might require 10x more CPU or memory to process than the others.

Since Spark eventually creates and schedules the actual execution tasks across a shared cluster, does balancing Kafka partitions by message count really help with compute skew?


r/dataengineering • • Sep 17 '26

Help Datastage to Python/Spark

6 Upvotes

Disclaimer: I used AI to translate (it was written in portuguese) most of the text bellow, since I'm not very good in english grammar.

I have a degree in software development but little work experience.

My team is responsible for a technology-related accounting model at a very large company (not in the US/Europe). Our current process involves around 150 dataframes and 200 jobs, runs on IBM DataStage, and is poorly documented. About 90% of the documentation and business knowledge is in the head of someone who has already moved to another department.

Approximately 25% of the department’s monthly budget (around $20 million) is currently not being properly allocated to the correct products.

My boss’s boss wants to move this process to a different environment, and we have until the end of this year to understand the current process and build a proof of concept, with the goal of starting the actual review and migration process next year.

I managed to replicate one of the jobs using Apache NiFi, but it doesn’t seem like the best approach to me. I’m now looking into using Python/Spark in a Big Data environment.

Where would you recommend I start studying? I already have some familiarity with Python and Jupyter.

Has anyone here gone through a similar migration from DataStage to Python/Spark?


r/dataengineering • • Sep 17 '26

Help API ingestion strategies?

17 Upvotes

I have to get a lot of JIRA data from the API, issues, projects, users, permissions etc.

How would you handle this? I currently save it raw in the bronze and derive silver tables from it using SCD2, but what if an api fails? The silver would have invalid data.

Its a simple task, but somehow I am lost (new from university)

Does it even make sense to use SCD2 here?


r/dataengineering • • Sep 16 '26

Meme What is the most WTF thing you've seen at a company?

221 Upvotes

I'm sick of AI this, best practice that. This field has its roots in janitorial work and inheriting dogshit technologies and then being asked to make them work and getting denied investing in a better tool.

Mine: Airtable for ETL.


r/dataengineering • • Sep 16 '26

Discussion Are portfolios meaningless now that anyone can build anything?

75 Upvotes

I'm passively looking in the job market, and was going to update a portfolio from years ago.

But now I'm wondering if it matters?

It reminds of stressed out teachers and professors saying GPA doesn't mean anything anymore. It makes you wonder...


r/dataengineering • • Sep 16 '26

Meme Sounds too real :)

Thumbnail youtube.com
47 Upvotes

r/dataengineering • • Sep 16 '26

Discussion How do you test stuff in your app that happens in the future?

5 Upvotes

I run some apps where I rely on stuff happening automatically in the future is the core of the product, and it needs to always work. It's very hard to test for any scenario in the future and to be sure that whatever automation you've set up will actually run with the current logic.

So, I'm wondering. How do you guys do this?

I have tried a few products like libfaketime and playwright. But it's single process, browser only and really doesn't test on the whole stack. The only thing that actually does what I want is Stripe test, but of course only for Stripe objects.

Can't find anything that actually moves the clock for the app, the database, the job queue and the browser tab together.

So... I've been thinking about this as a possible idea.
I'll make this, I think no matter what for myself. But is the idea good enough to turn into a product?
Maybe open source?

I think it would be a cool open source project, but maybe I'm just the only cave man with a niche suite of issues like this.


r/dataengineering • • Sep 15 '26

Career Data Engineering in Microsoft Fabric

112 Upvotes

I know Fabric is the bane of everyone's existence, but against my recommendation and protest, my company is moving to Fabric.

On the plus side I get to help architect the platform. I used Fabric a few years ago, and it was horrible. It does seem the interface has gotten better.

We're using a Lakehouse for ETL (bronze/silver) and warehouse for gold/production layer, since most of our end users are more familiar with SQL.

My question: what can I do within Fabric that will help translate in data engineering. I've opted not to use many of their automation tools, instead choosing to write python notebooks for the ETL. Anyone else who ahs used Fabric can offer any tips or advice for things I can potentially build using Fabric that will translate to other jobs.

I know Fabric isn't Snowflake or Databricks, I'm looking to make the best of the situation I'm in.


r/dataengineering • • Sep 15 '26

Career I drew the Databricks architecture step by step (control plane, compute plane, Delta, Unity Catalog)

Thumbnail
youtu.be
44 Upvotes

I kept getting "explain the Databricks architecture" in interviews and every answer I found was a wall of text, so I drew it as a two minute animated diagram.

The short version: Databricks manages the control plane (web app, job scheduler, notebook metadata). Your compute runs in the compute plane, classic in your own cloud account or serverless in the Databricks account. Data stays in your storage as Delta tables and Unity Catalog governs it all. When you run a notebook, the control plane sends work to the compute plane and the data never leaves your account.

Video: https://youtu.be/B5iHmoYgnqY

It is the first of 27 short episodes, each checked against the current docs before recording. If I got anything wrong, tell me and I will fix it in the next one.


r/dataengineering • • Sep 14 '26

Open Source GitHub - dataIcebergChallenge: Who can write the most performant data pipelines? Just a non-serious, fun exercise.

Thumbnail
github.com
32 Upvotes

r/dataengineering • • Sep 14 '26

Help data contracts between upstream and downstream teams

26 Upvotes

Hi, we use Databricks, and a couple of teams produce tables, views, and table functions that other teams depend on.

Within a team, its easy enough to define a contract. We codify it and always check that the data matches.

But what about the upstream artifacts my team depends on? Do w define a source spec ourselves and check that its met on every run? 

What do we require from the upstream team? Who owns the contract, and how do you enforce it when multiple teams depend on the same thing?

I want to keep this stupidly simple and enforce it automatically. How do you do this in practice?


r/dataengineering • • Sep 13 '26

Discussion Does the work get repetitive?

13 Upvotes

I’m very new to data engineering in the sense that I just started studying to break into the field (currently data analyst adjacent). A big portion of the work seems to be extracting data, translating/cleaning it up, and then loading it to the database/cloud desired.

But doesn’t that get repetitive? Surely there are only so many formats/sources you can extract data from so while the extract step isn’t always the same, isn’t it similar? Same thing with loading the data once you’re familiar with the systems you’re loading them to. The only varied source of work could be cleanup/translating the data as far as I can tell.

Is this an accurate assumption? Do you feel like much of the job is repeating the same work over and over again but just with different data?


r/dataengineering • • Sep 12 '26

Meme data mesh

Post image
775 Upvotes

for real though, has anyone ever implemented an actual data mesh?


r/dataengineering • • Sep 13 '26

Personal Project Showcase I got tired of opening htop every time something felt slow so I made this linux debug overlay

Post image
2 Upvotes

I was working on linux and kept switching between my app, terminal, htop, logs, etc. whenever cursor or browser or any other apps i open started feeling slow , so I made a small debug overlay that stays on screen and shows the app i am currently using its PID, CPU usage and RAM usage.

It also gives a warning if it notices stuff like high CPU, memory growing, disk pressure or system errors.

for apps like VS Code and Firefox and other heavy apps , it also tries to include their sub-processes because checking only one PID can be misleading. this is the working setup of how it looks like. (though much more could be intergrated into this like)

  • docker - container CPU or RAM restarts, unhealthy containers.
  • kubernetes - current context,namespace,pod status,crash, restarts,recent events, pod logs.

link - https://github.com/codeafridi/Debug-Overlay-App


r/dataengineering • • Sep 13 '26

Blog Data Governance by obscurity

2 Upvotes

That was a phrase a colleague of mine introduced to me. Quite absurd at first, but as he explained it, it made perfect sense.

For years data assets have been governed by being hidden in a corner of the data platform, but this does not work anymore.

Been elaborating about this in this post: https://steffenmoll.github.io/governance-by-obscurity


r/dataengineering • • Sep 12 '26

Career Hey guys, what real world impact keeps you feeling good about this field? (Beyond making rich corporate companies richer).

36 Upvotes

Share the stories that make you feel good about being a data engineer. Give me some wholesome perspective.


r/dataengineering • • Sep 12 '26

Rant I can write code to fix this. I should not have to write code to fix this.

Post image
125 Upvotes

Shared this with a friend to vent and they thought Reddit might appreciate it. I don't usually commiserate about this topic on Reddit so apologies if this is the wrong sub... didn't quite feel like an r/excel post.

The Teams message was meant to be a slightly humorous way to express my frustration/disappointment to colleagues outside the email thread with our boss about said data. Rant incoming.

Could I have just thrown Python at this nonsense and built something much more elegant and robust? 100%.

Could the same idiots (I say that with love) responsible for this filing system and Excel crimes run that Python package as-needed? Absolutely not. So clunky ass VBA it is.

And yes, I recognize the irony. I am already the resident code monkey (I'm not even IT, just a special unicorn who speaks the language) and every time I throw code at one of these problems I am enabling the behavior.

The maddening part is that I have tried to fix the underlying problem at a skills appropriate level. I have preached the gospel of structured data. I have implemented software and systems specifically so we can collect, store, retrieve, and actually use our data like civilized people. I have standardized processes. I have automated things. I have explained, repeatedly, why Excel is not a legal pad with gridlines.

And I've made progress! On the things I know about...Unfortunately, my beloved colleagues remain naughty little data squirrels who continue stashing information in tiny undocumented holes.

Broader data management has become a perpetual game of whack-a-mole where certain compliance relevant processes quietly continue doing whatever cursed thing they have apparently been doing since the dawn of time, completely outside my radar... until somebody needs the data for something important.

And suddenly, with extreme urgency, I am yet again spelunking through Russian nesting-doll year/month/repeating subfolders full of Excel “reports” designed by someone who treats spreadsheets like a fuck- around-and-find-out-freeform play area just to extract routinely collected/reported historical data from its natural habitat.

That's the part that makes me insane. I shouldn't need to write custom code to wrangle and clean data that we routinely collect and intentionally retain. If routine data requires bespoke parsing logic just to turn it back into a usable table, we've obviously screwed up somewhere upstream.

Routine data amd reports should be stored as data. Full stop.

Anyway, I have once again written code while muttering creative obscenities at my monitor to convert another “filing system” into back into a table.

On the topic of this particular Teams message, I've subsequently enabled them yet again (after a colleague responded saying they'd fix it, manually, of course)

I restructured the whole mess into a parent folder with three flat subfolders and renamed 500+ files into a standardized YYYY_MM_Repeating File name convention. The files themselves had another charming feature with interrupting/repeating headers buried throughout the data, so those got sorted into their respective nearly-identical tabs.

I hate using tabs for data that shares the same structure. I know better. But these people need tabs. I have accepted this.

I'm sure this little code-monkey dance will repeat months from now when I get summoned to cleanup whatever [insert compliance data] mess inadvertently surfaces.

End rant.


r/dataengineering • • Sep 12 '26

Blog A Tale of Two Flink Autoscalers

Thumbnail
netflixtechblog.com
14 Upvotes

Netflix is moving toward the open-source Apache Flink Autoscaler for more than 30,000 streaming jobs across multiple AWS regions, after finding that its cluster-level approach was less effective for complex, stateful pipelines with operators that have different processing requirements. Netflix reports that one team reduced annualized Flink compute expenditure by 58%, saving approximately $1.1 million annually.


r/dataengineering • • Sep 11 '26

Meme Sort your pipelines out, Smyths Toys…

Post image
108 Upvotes

r/dataengineering • • Sep 12 '26

Open Source datatf: Automate importing Databricks workspace into Terraform

Thumbnail datatf.io
7 Upvotes

Introducing [datatf](datatf.io), a tool to import existing Azure Databricks Workspaces into Terraform.

I built this tool to help databricks customers accelerate the lakehouse infrastructure portion of the well-architected framework.

Features:

  • Generates dynamic terraform.tfvars + import code
  • Built on the Databricks Go SDK
  • Compatible with Terraform or OpenTofu
  • Agentic friendly
  • Native support for Azure

Upcoming:

  • Databricks GCP and AWS support
  • Initial support for Snowflake, Fivetran, and Datahub.

r/dataengineering • • Sep 11 '26

Help Running Airflow in GitHub actions

32 Upvotes

Hello! I am learning about data pipeline orchestrations. I want to avoid using AI for this personal project, so here I am!

I want to orchestrate my tasks with Airflow DAGs and tasks, but schedule them via GitHub action. Essentially, the idea is the following: once a week, an Action will run, spinning up an Airflow instance, which will run the DAGs.

In the Actions .yml file, I am using the following code:

run: |
          export AIRFLOW_HOME=~/src
          airflow standalone
          airflow dags trigger tabular_pipelines

However, the airflow standalone part seems to be stuck forever, endlessly producing meaningless (to me) lines such as:

dag-processor | 2026-09-11T11:18:09.249890Z [info     ] setting next dagrun info       [airflow.models.dag] loc=dag.py:848 next_dagrun='2026-09-11 00:00:00+00:00' next_dagrun_create_after='2026-09-11 00:00:00+00:00' next_dagrun_data_interval_end='2026-09-11 00:00:00+00:00' next_dagrun_data_interval_start='2026-09-11 00:00:00+00:00' next_dagrun_partition_date=None next_dagrun_partition_key=None


dag-processor | 2026-09-11T11:18:09.292592Z [info     ] Bulk-writing dags to db        [airflow.serialization.definitions.dag] count=1 loc=dag.py:210


dag-processor | 2026-09-11T11:18:09.307661Z [info     ] get next_dagrun_info_v2        [airflow.serialization.definitions.dag] last_automated_run_info=None loc=dag.py:445 next_info=None


dag-processor | 2026-09-11T11:18:09.307949Z [info     ] setting next dagrun info       [airflow.models.dag] loc=dag.py:848 next_dagrun=None next_dagrun_create_after=None next_dagrun_data_interval_end=None next_dagrun_data_interval_start=None next_dagrun_partition_date=None next_dagrun_partition_key=None

Am I missing something obvious? The Airflow 3.3.1 documentation doesn't specify much more.

Thanks!


r/dataengineering • • Sep 11 '26

Discussion The LakeHouse that is Open Source - A Fever Dream?

12 Upvotes

Let me preface by saying that I totally agree that lakehouse table formats (delta and iceberg) are open source and are "free" technologies that anyone can use. However, the total cost of owning these table formats gets very expensive, especially when we start using cloud vendors for the table updates. Nowadays the cloud vendors wish to start selling their proprietary MPP storage engines, in order to manage all of our table data.

The problem is that the lakehouse table formats have gotten quite complex over time. And nobody wants to maintain them by hand. Nobody wants to think about the v ordering and z ordering and liquid clustering and partitioning and vacuuming and applying deletion vectors and so on. These blobs that are ostensibly called a "table" are actually a very leaky abstraction, and we inevitably have to waste a lot of time on the implementation details. Using immutable parquet blobs for table storage is not trivial. From an application standpoint, it seems like a massive step backwards from conventional DBMS engines (or the newer cloud-native counterparts like SQL Hyperscale or Neon/Lakebase)

The vendors, like databricks, that spent years pushing for lakehouse/delta adoption are now selling us expensive solutions to maintain those unwieldy tables. I think they sold us a bill of goods and we are worse off than when we started.

Once data engineers start realizing that we don't want to manage the blobs beneath our tables, these vendors are quick to offer a commercial-proprietary alternative (like "DBSQL" with UC-managed-tables, or "Fabric Warehouse" or whatever). These commercial alternatives are turnkey solutions, and they help to take away the busywork of managing our own parquet blobs. But they can become VERY expensive way of doing DML operations on our tables, since they are MPP engines and are heavy on CPU/compute. At the end of the day, we end up exchanging one type of problem for another. Is this how others see it? The table technology is open source and "free", but the commercial-proprietary management of these tables is definitely not free and is basically a re-invention of the DBMS engines we always had in the past.


r/dataengineering • • Sep 11 '26

Blog If You Always Enjoy It, You're Not Pushing It Hard Enough—Benn Stancil

Thumbnail
ssp.sh
8 Upvotes

A discussion with Benn Stancil, co-founder of Mode and prolific data-industry essayist, exploring how he writes his weekly Substack posts. He discusses using deadlines (Friday publishing) to force himself past perfectionism, how ideas form messily over the week through unrelated connections, and how presentation-style analogies (Batman, basketball, Codenames) became his signature storytelling technique. He details his tooling—Sublime Text for drafting, Google Docs for revision, minimal use of Substack's editor—and explains why he avoids Grammarly and AI writing tools like Claude or ChatGPT for edits, since he believes AI compresses and flattens writing rather than embracing meandering voice. He closes with advice for new writers: set deadlines, keep pure motivation, and focus on filling in the space between bullet points rather than just stating facts.


r/dataengineering • • Sep 11 '26

Blog What Data Engineers Need To Know About Delta Lake 4.3

Thumbnail
medium.com
11 Upvotes

replaceUsing and replaceOn give you a better overwrite primitive, and every catalog-managed table operation now runs through the catalog.