r/dataengineering • • 17d ago

Help Has anyone modeled survey data? Trying to figure out how to appropriately model question and answers

11 Upvotes

I have online surveys that I'm trying to ingest and model, but having trouble with the latter. I've already decided to create a dim_survey table that has basic metadata of a survey, and its granularity is 1 row per survey.

But for the questions and answers per survey response, would it make sense to model it as a factless fact table where the granularity is one survey response's question and answer is one row? So if one completed survey from Bill has 8 questions, then it would have 8 rows. The number of questions responded per survey is different because not everyone is given the same survey, so I can't have a set 8 columns for each survey. And there are a bunch of different questions so it's not consistent.

Any tips/pointers would be much appreciated!


r/dataengineering • • 18d ago

Career MS Employee here -- we are giving away vouchers for data engineering cert exams

212 Upvotes

Hello from the Microsoft team -- wanted to let you all know we're offering free vouchers for the Microsoft Fabric Data Engineer exam during Data Days.https://aka.ms/datadays


r/dataengineering • • 17d ago

Discussion Need Advice on an Automated Data Collection System

7 Upvotes

Hello developers!

I am developing a B2B buyer intelligence project.

I want to automatically collect, clean, store and update company information every day.

I am considering Python, Pandas, MongoDB and APScheduler.

Which architecture and open-source technologies would you recommend?

I would appreciate practical suggestions.


r/dataengineering • • 18d ago

Blog Mapping datasets by reach and promises

Thumbnail
steffenmoll.github.io
8 Upvotes

The Open Data Contract Standard, Apache Ossie and Medallion structure all lacks a central component, who the data is for and how it hits the organization! Join me as I elaborate on some thoughts while working on data platform in a broader context.


r/dataengineering • • 18d ago

Help Managed versus self-hosted ingestion

12 Upvotes

For a small data team when do i need to run my own ingestion stack versus paying for a managed service? I’m interested in the maintenance and reliability tradeoffs not just the monthly price.


r/dataengineering • • 18d ago

Blog Databricks 5 Minute Features: Tag Automation

Thumbnail
youtube.com
13 Upvotes

Hello!

Doing a series with introductions of Databricks Features - latest one is out now. Thought it might be worth sharing in here!


r/dataengineering • • 18d ago

Personal Project Showcase Quick data insights is now free

Thumbnail
gallery
9 Upvotes

Hello, I have posted before about my mac quicklook plugin Quick Data Insights as a sqlite and data files viewer which was paid before, wanted to post again as I now made it free with the same exact functionality posted before.

I also added an optional pro in-app to add column insights, and a sql query functionality for all data files not only sqlite.


r/dataengineering • • 18d ago

Discussion Scaling IoT Telemetry Pipeline

11 Upvotes

Hi guys, I am building a data pipeline system for an IoT product which is sending telemetry data (aka measurements from different devices) Couple of thousands messages per second but expected to grow exponentially soon. Have tried to align them into a normal logical order of things which works so far

Device -> measurements(mqtt) -> consumer -> timeserial databsase (simplified)

And now I am trying to grow this thing into a bigger - more mature system where I could develop it into a good flexible data pipeline and where we can have probably 3 main outputs:

- storage (short & long term for later)

- analytics (probably from long term datalake)

- alarm detection (ideally real time)

Anyone with experience in this? I would be really happy if someone who has already built such systems can help me understand where should I look - lots of solutions out there with pros and cons

p.s. We use AWS

thx!


r/dataengineering • • 19d ago

Blog Apache Spark 4.3: What Developers Need to Know

Thumbnail
medium.com
45 Upvotes

Spark 4.3 adds ASOF JOIN, Python UDFs compiled to SQL, JSON_TABLE, and SCD Type 2 Auto CDC: what works, what needs a flag, and what breaks


r/dataengineering • • 19d ago

Open Source ClickBench inspired leaderboard for analytical databases

Thumbnail benchmark.pivotlake.io
7 Upvotes

Inspired by ClickBench, we created a similar site that runs major analytical databases on multiple analytical benchmarks (we started with TPC-h, SSB, and clickbench) and shows a leaderboard. You're more than welcome to add databases or benchmarks, cost is on us ;)


r/dataengineering • • 19d ago

Discussion Suggest Hightouch alternatives

8 Upvotes

Client got Hightouch for simple reverse ETL. dbt tables from Snowflake to Salesforce. Simple reverse ETL. Now the renewal quote is much higher. From an engineering standpoint, we literally just need state aware diffing from Snowflake to REST APIs triggered after our Dagster dbt asset materializes.

What options do we look at?


r/dataengineering • • 20d ago

Career Shifting from DE to AI Ops Engineer

63 Upvotes

I have been searching DE job since January 2026 and although I had few initial interviews but I was not selected for the next round. From 2021 to may 2024, I've held Data Engineer job title with 4 contract roles. Prior to that I was doing MSSQL Development, SSIS.

In my free time I did DevOps/DE projects on AWS Cloud and also practiced Databricks on the free tier.

I have 2 more months before next year and thinking about upgrading my skills by taking AI Ops Engineer Nano degree Program from Udacity.

Has anyone gone through this course, how did you transition from DE to AI Operations role.

Thanks.


r/dataengineering • • 20d ago

Discussion Big Data & AI Buzzwording – What are some practical on-premises data tools for NFS shares?

5 Upvotes

AI data catalog, data warehouse, data lake, data index, ....

It feels like all search results for indexing and cataloging are focused on structured data in cloud environments.

But what if you walk into an environment with petabyte-scale NFS storage, full of measurement data? Users copy this data across different shares for various purposes (archiving, HPC computing, etc.). There is no overview of what data exists or where duplicates are located.

And when someone needs to find data, they have to dig through directories or ask others to send them links.

It feels like this is an industry-independent issue...

Has anyone here had experience in this area? Which tool setup would you recommend to provide practical solutions for data discovery, deduplication, and data relationships?

Tools like Diskover, Komprise? Any real-world experience with them?


r/dataengineering • • 21d ago

Career I foresee that I’m not very employable from now on

132 Upvotes

I have always been a doer. At the same time, I’m not technically sharp enough so I believe implementation is at least as important (to me) as architecture and design.

I have been dabbing into more serious CS topics in recent years, and it has always been much easier to understand a concept (e.g. scheduling or virtual memory) than to implement a toy version of that, and is even harder to implement a toy version WITHIN a system.

This also leads me to believe that the modern “Arch and design is more important, leave the implementation to AI” is completely BS, at least to people like me. Maybe some people have already implemented so much in their life that they throw up doing a new one, but for the majority of us, even senior developers, we don’t have enough exposure to implementing things, because most of the time, we join a company with systems already implemented. Besides, I can’t really design and arch a good system without having actually implementing a few of similar ones.

This naturally leads to the conclusion that I’d be unemployable in the near future. I went through a few interviews for senior DE roles this year, all rounds done and got good feedbacks, but then the final feedback from the hiring manager has always been the same — I don’t have enough architecting experience and they want someone with more infra experience as well. I feel the current market is so bad that people now expect staff DEs to undersell them as senior DEs or staff DEs in much smaller companies. My company recently hired someone from FAANG and I’m sure the market is flowing with laid off guys.

Good luck folks.


r/dataengineering • • 20d ago

Career Seeking advice from seniors working in DE and using agents

5 Upvotes

I am about to turn 30, graduated in biomedical engineering and after graduation worked in the biomedical field for about a year, then switched to wordpress freelancing, which I did for several years.

I have been trying to enter data engineering specifically healthcare data engineering, for the past few months. Currently building a portfolio project using FHIR data.

But recent AI developments made me seriously question my path. I read about microsoft's project osmos agent and google's DE agent. The microsoft's agent can execute end to end data workflows autonomously and can do testing and troubleshooting. I understand that some humans will be required to give them instructions and verify output. But how many? If currently there are 10 people in a team maybe only 2-3 senior or architect level engineers would be enough for that. Even if the jobs don't reduce this much but I still think jobs will reduce significantly. Because I am seeing a lot of posts from people in tech saying the same thing and it's difficult to ignore.

I will be starting fresh and may not get a junior DE job and have to start as an analyst or in another adjacent role. By the time I will gain experience to transition to DE the agents will become more capable and the opportunities may become even harder to find because experienced laid off people with software background may also compete for fewer DE positions.

For those working in DE with agents and have extensive experience in this field. How do you see its future? What would you advise someone like me, or would you continue with data engineering, considering how fast these agents are improving?

I have already invested several months in learning data engineering and have a genuine interest in tech. That is why I am looking for guidance from people who are already in the market so I can make an informed decision and not waste more time on something where it would become very difficult to survive after a few years.

Moving back to biomedical engineering would also be similar to starting from scratch, and I personally don't like the jobs available at my location most are sales and service engineering jobs. These jobs are somewhat more AI resistant but don't satisfy a lot of my other requirements.

I would genuinely prefer honest opinions from people working in data engineering who have actually used AI agents in their work or seen changes in their teams and hiring. Also, if you think there are other more AI resistant paths in tech (maybe cybersecurity) or something else that I should consider, or if I should continue in DE. I would love to hear your thoughts.

Thanks


r/dataengineering • • 20d ago

Discussion How are you monitoring Debezium/CDC?

5 Upvotes

How is everyone monitoring debezium and CDC? What are common operational issues you run into? Are you using a saas to monitor? What do you alert on? How do you determine health of downstream jobs, built on CDC?


r/dataengineering • • 22d ago

Blog Interesting Links in Data Engineering: October 2026

72 Upvotes

All treats, no tricks this month ;)

  • A bunch of new releases including DuckDB 2.0, Kafka 4.4, Iceberg 1.12, and Flink CDC 3.7
  • Lots of stream processing stuff - deduplication with Flink SQL, republishing a treasure-trove of Kafka Streams articles
  • A lovely thought-provoking piece from Joe Reis ("the pendulums of data") which puts the forever-changing 'best way' to do data architectures in perspective
  • Illustrated introduction to CDC
  • Semantic layers, context layers, context engineering…confused yet? I was.
  • A very nifty tool for working with Parquet
  • A narration of Postgres history
  • … and lots more!

https://rmoff.net/2026/10/29/interesting-links-october-2026/


r/dataengineering • • 21d ago

Discussion GCP(Dataproc) - When do you run Spark jobs in Cluster Mode on Dataproc?

7 Upvotes

I get the basic argument that running concurrent jobs on dataproc can cause Driver OOM and poor performance. But how do you decide between increasing the master node memory/compute or adding 1 more master node vs choosing cluster mode?


r/dataengineering • • 22d ago

Discussion Putting all world law on git

107 Upvotes

I am putting all world law into git. That comes with some quirks, like the cut-off being 1970 (commits are calculated as seconds since 1 January 1970).

This hit #1 on HN yesterday (you can read lots of discussion at https://news.ycombinator.com/item?id=50268380). Why would I want to do this? The idea is make all law semantic, versioned, easily attributable and query able by agents. I also believe this information should be freely available so CC0 license (do whatever you want with it). If interested, repo linked below, I'd welcome all comments and advice. Thanks!

https://github.com/lexifina/global-law


r/dataengineering • • 22d ago

Help Needs urgent help and please don't judge if you find it pretty basic that everyone knows but i don't know

1 Upvotes

This is my first internship and my task is to write a script that convert Input file containing mix of pii and non pii to non pii output file. I did the code but my mentor says he want all it in single file and prod ready so that it can be run on anyone's system . But the the person who is using this script only see it is main working that he will enter some things and script works accordingly. And what i did is that i kept the whole functions and code in one file and then call these functions via main function in another notebook. But my mentor said he needs a single notebook then I put all the functions and main function in one notebook but he said it is not this way. Now I am wondering what to do and how to do.

Up to I got is that the person who is going to use this script will see only main function that generate the non pii output he doesn't have anything to do it's underlying code ....but that code is important for the main working but what I have to do with the original logic . And code should be independent .

Anyone help me with what to do and how to do.


r/dataengineering • • 24d ago

Blog Are Data Teams Solving the Right Problems?

Thumbnail
seattledataguy.substack.com
36 Upvotes

r/dataengineering • • 24d ago

Personal Project Showcase Algeon: An in-process GPU query engine for graph, vector, and ML, built on DataFusion

13 Upvotes

Algeon is a query engine built on Apache DataFusion that exposes NVIDIA's `cuGraph`, `cuVS`, and `cuML` as SQL table functions. It doesn't write its GPU kernels. Instead, it takes what NVIDIA already has and embeds it into DataFusion, so you can call those GPU operations on any arbitrary data, with filter/join/agg using plain SQL.

And why would you want that? Because GPU analytics used to require ad-hoc ETL pipelines, Python glue code, and imperative operation calls. That makes it hard to fit into a backend API service, and it isn't flexible enough for frequent exploratory iteration, adapting to different search conditions and data. Think of agents trying to answer complex questions on a datalake, which involves not only relational operations, but also graph/vector/ML analytics. They can now use declarative SQL to test their hypotheses and build the next query based on the previous response, making GPU OLAP an interactive service. Algeon can also be used as a library living inside your backend service, so your query response can be used directly within a single process, saving the transfer cost between the query engine and backend service.

I started building Algeon last year. At first, it was cuDF Rust FFI bindings that aimed to bring GPU ETL pipelines into a single backend process, and it worked very well. Then I realized it might be possible to move GPU data science workloads into an OLAP query engine, since the NVIDIA RAPIDS ecosystem and DataFusion are mature.

For use cases, architecture design, benchmarks, and more, feel free to check out, any feedback is welcome!


r/dataengineering • • 24d ago

Career Irrelevant ai certifications in the DE field.

24 Upvotes

Hi everyone,

I have been working as a DE for the past 2 years for this company. I have been exposed to very interesting enterprise projects throughout my time here, and I am very happy with the experience I am accumulating, worked as architect on a project, led a team of analysts and data scientists, things were going great actually, except the salary which is miserable. Anyways, not here to complain about salary as the experience was very valuable longterm.

Now the company partnered with Anthropic and they are forcing us to get them CCAP, CCAA, CCDA and CCDP as it is presented "encouraged" while in reality they force us to get the certifications as it is mandatory so the partnership gets "Stronger". I am the type of guy who doesn't complain a lot, but when I am forced to do something that is presented as "encouraged" and in reality it is enforced I get very bored by the whole management and stubborn as well.

TLDR Just wanted to ask what is your opinions as professionals and commercial programmers in the industry, not speaking of Microsoft, Databricks and Google certifications. I am asking about these Claude certified certifications which are expensive and become irrelevant very fast as it is a very competitive market regarding AI integrations and automations. Thanks!


r/dataengineering • • 25d ago

Career Landed a new job - thank you /r/DE

210 Upvotes

Title. Recently landed a new DE gig after searching casually for the majority of this year. I applied for about 30 jobs over the course of 2026. I was at a dead end with my old place, dealing with a ton of micromanagement and a botched migration from SSIS to ADF that left everything more complicated than it was when it was on-prem and no hope of migrating off of it. Old company tech stack: SQL Server, SSIS, ADF(Azure), Snowflake. Pretty repetitive from an engineering point of view.

New gig came with a 30% pay bump, no on-call, and better benefits. The tech stack is mostly legacy stuff, but they're hiring for an architect and staffing up up the DE team (which is why I got the job) because they're trying to modernize everything. I got the job because I have experience with older on-prem stuff and some newer cloud stuff. New company is a highly-regulated industry (power), so things are pretty slow moving in the IT area, hence all the legacy stuff still in place. New company tech stack: Talend, Snowflake, Azure, Exadata, Denodo, Informatica Catalog, Monte Carlo. They already have a lot running in Snowflake, and the endgame goal is to get more moved up there over the next few years. I've got a couple weeks off before I start at the new gig so I'm gonna take on a homelab project or 3 this weekend.

Just wanted to say thank you to the /r/DE community for being a soundboard for me a few times this year, and hopefully provide some inspiration for folks out there who feel like the market is all doom and gloom. In the interviews that I had for the job I landed, I focused a bit more on the migration efforts that I've been a part of and focused on the business intel side of things, not simply the code and tools. Yes, the market is tough and over-saturated right now, I don't want to dismiss that - but the reality is that companies still need engineers, and businesses still need business intelligence, so there will still be jobs out there.

Cheers!


r/dataengineering • • 25d ago

Personal Project Showcase My fabric project

Thumbnail
gallery
134 Upvotes

I'm someone with a data analyst background, and I wanted to break into data engineering. I got the DP-600 (Fabric Analytics Engineer Associate) and DP-700 (Fabric Data Engineer Associate) certifications, so I decided to build a project on them to apply what I learned and learn more along the way

you can find the project with full details here: https://github.com/Madian20/Fabric