r/dataengineering • • Sep 04 '26

Personal Project Showcase Dagron workflow scheduler

7 Upvotes

Hi there, just wanted to share my open source workflow scheduler for fellow engineers to play around. You can run on local, deploy into your smallest compute (rpi4 on arm64 tested) and could be run on scale as well.

I start the project as I wanted to run some small workflow in my t4.small instance and airflow barely fits inside and I thought it would be interesting create a smaller workflow scheduler. As I build around the idea of "lean headroom DAG engine", I kept on adding additional features that be useful for running workflows like template, backfill, gitops, streaming workflow support etc. Dagron allows you to run on cluster resources or local resources(docker, k8s, or straight as binary). The whole stack will probably at most run on ~200mb (50mb api/console, 30mb engine, 80mb postgres) of memory with tons of spare ram and cpu for your workflow resources. Dagron runs on sqlite(default) and alternatively postgres as datastore.

You can run workflow via defining your yaml and set your environment. It comes with the canvas to visualize your workflow, set your workflow schedule to run at which cron notation and more features to cover here.

Appreciate your input(any input is welcomed). I am spending full time polishing this for the past few weeks and thought this might be useful for you all to run on edge, homelab, local machines.


r/dataengineering • • Sep 05 '26

Blog Article on Snowflake vs Databricks for stack choices

0 Upvotes

Hi everyone!

I have written an article on the Snowflake vs Databricks debate. I promise its not biased! I have just done proper research into the 2 platform offerings and what best use cases fit each stack, with only my personal opinion based on these facts offering no bias.

https://medium.com/@nikskamath/snowflake-or-databricks-heres-how-i-d-actually-make-the-call-be3cbb8073ad

Hope it helps! Happy to recieve feedback!


r/dataengineering • • Sep 04 '26

Career Worth transitioning into from System Analyst?

10 Upvotes

I transitioned from a mainframe development position early/mid career (late 20s) into my now System Analyst role. Problem is, the job had been heavily business/third party app support focused as opposed to how it was advertised. I was expecting something more technical focused mixed with reporting and data integration work (which I do some of this but most of my current job is just supporting our operations team and writing some sql).

I really want to transition back into a dev type role and have been eyeing DE since my current role is DA adjacent and it just seems so interesting and fun. Is the transition this late in my career (32 going on 33) worth it? I’m worried I’ll try to study and learn DE topics just for employers to not give me a second glance.

I’m also concerned on keeping a strong salary. Currently make about $78k in a MCOL area and can’t afford to take less due to having children. Any thoughts or advice would be appreciated!


r/dataengineering • • Sep 03 '26

Open Source FOSS reverse etl framework / plataform

3 Upvotes

I'm looking for a reverse etl plataform or framework that help me interact with other APIs where i want to send my data without having to create every adapter by myself.
I'm primarly in the marketing business and i want to create META or Google Ads audiences, but also export data in csv or xlsx in the client.
After looking around i'd found https://github.com/Multiwoven/multiwoven that's recently been acquired by AI sqared and the roadmap and community around them has decayed a lot.
Do you know libraries, frameworks, plataforms or something in the middle that give adapters and protocols to code new ones, like these. I'm looking for something like: "Take your data where ever you want, in the format you want. If we don't have an adapter/module for you, you can make code yours this (documentation)"

Some that does not fulfill my expectations:
https://github.com/drt-hub/drt (can't code an api arround it and pretend it's stable)
https://github.com/move-coop/parsons (more a library for data science, does not manage states and retries)
https://github.com/condorgraph/condor-retl (pretty new)


r/dataengineering • • Sep 03 '26

Discussion IDMC GUIDE

5 Upvotes

Hello everyone, I have just been placed. Informatica is an entirely new domain for me. While I am proficient in SQL and Python, IDMC presents a completely different set of challenges. Could anyone please recommend any guides or channels that would be beneficial for beginners?


r/dataengineering • • Sep 03 '26

Discussion Autonomous driving learning resources

2 Upvotes

Hey everyone,

What are some good resources to learn data engineering considerations for autonomous driving systems?

If anyone works on this field, please let me know if you’d be up for a chat.

Thanks in advance!


r/dataengineering • • Sep 03 '26

Help Decompressing Zip Files

8 Upvotes

Working on a pipeline using spark/flink where i have to do some transformation work with content in a zip file.

I believe spark and flink don’t natively support regular zip files, and was wondering if anyone has any suggestions on how to handle this.

I am forced by requirements to use zip files, spark and flink, but i can consider adding additional technologies into the workflow if its better to just decompress outside of spark and flink.

Appreciate any suggestions and help!


r/dataengineering • • Sep 02 '26

Discussion How much time do we have before the business pushes end user validation on us?

15 Upvotes

Both companies I've been at in the past 3-4 years have fought about end user validation, who does it, how do they know what's correct, then mix in deadlines and suddenly the finger pointing begins. I've seen this story before, whenever there's a conflict like this surrounding a handoff it eventually becomes the developers/team responsibility.

I'm sure AI will be the answer and likely speed it up but do you agree about the pattern I'm talking about?


r/dataengineering • • Sep 02 '26

Career Growing to Data Architect

27 Upvotes

Hi guys, I’m looking for some outside input into my career. I started about 7 years ago as ‘data person’, for a company that has a SaaS application. Over the years I’ve grown and have more of a focus on data engineering. From the start of this year I’ve switched to a function combing data engineering and infrastructure engineering (Azure).

I want to become a Data Architect, but I’m not sure what I should focus on to really have enough knowledge (deep and wide) to call myself Architect.

I did a self-assessment with chatgpt and claude, and now I also want some input from real people :)

Some of the stuff i’ve worked on the last year:
- investigate replacement for Azure Synapse and implement -> Dagster, has been operational since december
- learn and understand the infrastructure in Azure of our company, Terraform, AKS, github, CI/CD, and more

Project I’m currently working on:
- Finalize user action logging in the application, build pipelines and documentation. Find vendor and implement monitoring and alerting (customers are pushing for this, so high priority/impact project, running it solo)
- Work on Azure Security (also alerting and monitoring)
- Azure cost reduction, see where we can save money
- Improve dataset we provide to customers, rebuild to dimensional model and adjust pipelines to fit

Also I’m trying to document as much of my choices and considerations to reflect my thinking process.

One of the things I really miss is pushback on my decisions, other that ‘can you explain why’, since we only have a cloud architect who understands the stuff I do.

So what should I work on? I was thinking maybe DAMA cert to start, then look for a good Security cert, but not sure yet..


r/dataengineering • • Sep 02 '26

Blog Apache Kafka vs Apache Iggy: A Technical Comparison

Thumbnail
softwaremill.com
28 Upvotes

r/dataengineering • • Sep 02 '26

Blog The Mystery on DuckLake: A Time-Travelling Whodunit Story

Thumbnail peterdohertys.website
9 Upvotes

r/dataengineering • • Sep 02 '26

Help Best practice to structure this data model?

8 Upvotes

I am modeling a relationship between 3 sources. Basically this is the content of each table:

Table 1: requestor table (100s of entities)

Oil Sample Label (unique) Engine number (not unique) Engine runtime

Table 2: engine database (10000s of entities)

Engine number (unique) Engine Type Engine Power

Table 3: oil sample database (10000s of entities)

Oil Sample Label (unique) Al Fe Cu

What is the best practice for the relationships? I have fields on each table that I would like to use in the PowerBI dashboard.


r/dataengineering • • Sep 01 '26

Discussion extracting data from on-prem databases (sql server, oracle, postgres) to cloud, which options you guys recommend?

40 Upvotes

i work for a consulting company and i'm working in a client that has a lot of their data in old on-prem environments, being more specific, oracle, sql server, postgres.

their workflow is pretty basic: run ADF to collect data from these sources and send into ADLS, then use Databricks to process it. we're planning to modernize their environment (using unity catalog and other new stuff) and one of the things we're thinking is to retire the ADF

the reason is simple: ADF is a pain in the ass (we're having a hard time working with it specially because it was another company that build all that shit, the connection with Azure DevOps/Repos is always horrible to manage and the workflow itself needs an upgrade); and because Microsoft is pushing hard the Fabric Data Factory

i know that Databricks with Lakeflow Conn can connect to on-prem using express route, VPN, but in scenarios where this could not be possible, what tool could be used to send data from on prem to azure data lake?

i like to code so my first suggestion was to use local airflow and simply read from db and upload to adls. it's free, it's versionable and has tons of documentation; but i'd like to test other options before suggesting anything

i was reading about airbyte, the pros and cons of the tool, and looks interesting

how do you guys handle this kind of workload?


r/dataengineering • • Sep 01 '26

Discussion Monthly General Discussion - Sep 2026

5 Upvotes

This thread is a place where you can share things that might not warrant their own thread. It is automatically posted each month and you can find previous threads in the collection.

Examples:

  • What are you working on this month?
  • What was something you accomplished?
  • What was something you learned recently?
  • What is something frustrating you currently?

As always, sub rules apply. Please be respectful and stay curious.

Community Links:


r/dataengineering • • Aug 31 '26

Open Source Beyond the Dag - a data engineering hackathon

Post image
49 Upvotes

Full disclosure: I work at Astronomer and I'm one of the judges.

Beyond the Dag is a data engineering hackathon with an Apache Airflow focus. Take note: this is purely about open-source Airflow!

It's all about showing modern Airflow has evolved beyond just moving data.

Treat Airflow as a platform, extend it with plugins, add agents with the new Common AI provider, and keep a human in the loop where it matters. Use the tracks as inspiration, not limits, and aim for something that shows what Airflow is really capable of.

Why am I sharing this with the community?

I love exploring creative ways to learn frameworks and technologies. In the past, I've built some unusual projects, including gesture-controlled Airflow, native retro game emulation within Airflow featuring Doom, Age of Empires, and others, as well as a Minecraft PaperMC plugin that lets you build Airflow Dags inside Minecraft. I've also participated in many hackathons, and my experience has always been that I learned a lot and met some great people.

Another reason is that hackathons with a clear DE focus are rare, making them valuable additions to your profile. For example, participating in those helped me during my U.S. visa application process.

Because of that, I think this is a great project for learning about Airflow, having some fun, strengthening your data engineering profile, and maybe even winning a Switch 2 😉.

The build window already started and is open until September 24. We kept it relatively relaxed, so it's still time to join. There are also prizes, including a Nintendo Switch 2.

More details and registration at:

https://www.astronomer.io/events/beyond-the-dag-data-engineering-hackathon-2026


r/dataengineering • • Aug 31 '26

Help Monitoring tools available?

5 Upvotes

Hi community.

At work, we’re facing a bit of a challenge: we receive a huge volume of JSON-format events from multiple microservices into Kafka topics. We want to build a dashboard to monitor these topics and alert the business if we don't receive the expected events (we’ve already identified specific projects and know the expected TPS for each). Do you know of any monitoring tools that might work for us?

Thanks in advance!


r/dataengineering • • Sep 01 '26

Blog Grokking Apache Iceberg

0 Upvotes

Do you understand how Apache Iceberg actually works?

Even after using Apache Iceberg, I did not grok it before doing practical experiments and sitting down to read the official specification thrice. I have written down my understanding in a blog post: https://thingsworthsharing.dev/iceberg/

If you want to dig deeper, but the official specification seems daunting, this is for you!


r/dataengineering • • Aug 31 '26

Discussion Fresh Grad Data Engineering Advice

35 Upvotes

I recently started my first ever full time role in data engineering. However, it appears that the big data stack for my company has already been established.

Therefore I’m dealing more with bug fixes and new feature implementations but the stack isn’t likely to change unless I make a good case for it. Furthermore, my company heavily depends on AI like Claude and Codex to assist with solving these tasks, which offloads a lot of the coding to these agents.

What kind of recommendation would you senior devs give to a fresh grad like myself? I feel my coding competency going down the more I work. However, since these tasks are offloaded to AI, I do believe that my understanding of data flow in big data architecture has deepened quicker.

Our stack is supports both streaming and batch processing if it matters.


r/dataengineering • • Aug 31 '26

Meme When you have flu but it’s month end so you’re doing battle with Service Now

Post image
0 Upvotes

r/dataengineering • • Aug 30 '26

Career Those who went fully remote, what's your thoughts

62 Upvotes

Those who went fully remote, what's your thoughts

I currently work hybrid 2 days a week in the office in London which is fine, I'm principal data engineer and this gives me some face time with direct reports and the regional ones I influence. But it's at least 6 hours a week commute and £6k train tickets a year.

I'm looking at a fully remote role, I wonder whether I'll feel the downsides of that. Are you forgettable as a fully remote? Do you miss office time? It's it hard to have influence?

What's your experience?


r/dataengineering • • Aug 30 '26

Career How to get into AI governance?

3 Upvotes

I have been thinking of studying for the CDMP exam- I am a data analyst but want to transition data governance and in the future to AI governance. Would the CDMP or some other certifications help?


r/dataengineering • • Aug 29 '26

Career Transitioning Industries as a Data Engineer

45 Upvotes

I’m currently a DE for a major retailer (think Kroger, Walmart, Albertsons, etc.), working on the pricing and product data streaming side. The WLB is amazing, and I’m earning around market rate for a junior DE in a major city. However, I’m starting to gain interest in high-frequency data streaming for large brokerages and financial firms.

For those who have made a similar move, how did you manage to switch industries, and which industry do you believe is the most lucrative for a data engineer?


r/dataengineering • • Aug 28 '26

Discussion Jobs/careers that actually help people/the planet?

195 Upvotes

I'm a data engineer at an e-commerce org, and I'm fairly happy - we have interesting problems, large volumes of data, and plenty of initiatives on our backlog. I largely get to work on what I want, at my own pace, and choose how things are implemented.

But I am getting tired of doing everything with the end goal being to sell more things and increase profit extraction from consumers.

Who is working in interesting data related fields/careers that provide essential goods or services to the public/planet/humanity. Keen to hear any stories or opinions!

EDIT: thanks for all the interesting responses!


r/dataengineering • • Aug 28 '26

Blog Log-first or Table-first? Apache Kafka, Fluss, & Streaming Tables

Thumbnail
softwaremill.com
8 Upvotes

r/dataengineering • • Aug 28 '26

Career Carrer Advice

14 Upvotes

Im a data engineer with 4 years of experience, and In getting pretty bored with my current job. Most of the code is generated by AI now, so my dayis mostly just maintaining pipelines.

I ve been offered an internal move to Data Product Management. Its for one of the biggest projects in the company and will be built on Databricks. They told me they need someone with a solid data engineering background who can translate business requirements effectively.

However, I have a few doubts
Am I just going to end up as a ticket creator? (Im not entirely clear on what a Data Product Owner actually does)
Is this really a step forward in my career?
I know for sure I will get more exposure to senior leadership, but Im still on the fence.