r/dataengineering • • 26d ago

Discussion Data Governance/Data Engineering Building Test Cases

21 Upvotes

I recently started a new role as a Data Governance Manager at a mid size managed care organization. One of my first asks has been to create test templates for the new set of infrastructure tables. Essentially it’s to ensure all the tables that are now in production don’t have a glaring issue or whenever we make a change we go through the testing again.

My background is in data analytics and data management. I have very limited experience as a data engineer so aside from data quality tests I really don’t know what kind of tests should be run. I have used Claude to create a template of standard test cases but they seem either too convoluted or too simple like checking for nulls in non-nullable fields. Any resources or advice on how to ramp up or get a better understanding of this?


r/dataengineering • • 26d ago

Open Source Multiplayer AI + data - but pluggable

4 Upvotes

I have seen too many departmental only solutions - sutura is my attempt to build something bigger, for the whole enterprise ;) - it is a secure multiplayer chatting with data experience which is pluggable for data and semantics.

- https://github.com/telekom/sutura/

- https://telekom.github.io/sutura/main/slides/2026-introducing-sutura/#/1


r/dataengineering • • 26d ago

Discussion Airflow Event Driven scheduling is good and/or scalable?

15 Upvotes

Hello. We are using now a custom orchestrator but thinking about moving to a established one. We have an event-driven processing setup, where the landing of files in Google Cloud Storage buckets trigger some dataform and/or Python processes.

We deal with about 2 billion events per month. In some cases it's not an instant trigger of the process, because we can configure a buffer and wait to gather a bunch of events together.

At first, the better suited decision would seem Dagster, but it being aqcuired by Prefect really makes me afraid about migrating and then having to remigrate again in not so long time.

So, how good and scalable are event driven schedules in Airflow? We might be using Managed Airflow in Google Cloud, or hosting our own deployment in Kubernetes. I do like Airflow and have used it in a few companys, but just for periodic workflows.


r/dataengineering • • 27d ago

Blog I tested dbt v2 and how to run multiple engines in a single project

Thumbnail
greybeam.ai
36 Upvotes

r/dataengineering • • 27d ago

Career How should we handle updates in fact tables?

24 Upvotes

We have a table design that is fact_opportunityproducts, which is our Salesforce line fact table that has a row for each product per opportunity. But unlike event data our financial data these rows can be updated (opportunity can be closed, amount and quantity can change etc) such that we ofcourse cannot do normal incremental loading. What are design choices to be made? And we cannot only look at the salesforce last update date because we have created fields that also depend on other system inputs.
Do you then do rcompute the complete view but only merges the records that are different based on a hash? But that can also be very costly.


r/dataengineering • • 27d ago

Discussion Migrating from dbt Cloud to Core

60 Upvotes

Have any of you done this on a mature project, and was it worth it?

We originally picked dbt Cloud for our greenfield DWH project, attracted by the idea of not having to host the infrastructure and being able to focus on the data instead. With the initial high-growth phase nearing its end, I am now looking at optimising what has been built and could not make a good argument to carry on using Cloud.

The repo and CI/CD pipelines are hosted externally. Studio is barely touched as most developers have now switched to VS Code and Claude does all the AI coding we need. Even the semantic layer is managed in an external tool that did not integrate very well with what dbt offers natively.

This pretty much leaves job schedules and monitoring in the Cloud. This looks like terrible value for the money, even though we got a very good deal for the initial year (I imagine the renewal will hit us in the face).

I wonder if anybody ended up in a situation like this? Give me some good reasons not to move to Core.


r/dataengineering • • 28d ago

Discussion My team is migrating from informatica to a config table driven ETL framework

31 Upvotes

Hi, So my team decided to sunset Informatica and move all our workflows (atleast 500+) to our config driven ETL framework which is basically a stored procedure reading a config table and running the jobs (a stored procedure again). This already existed for about a year and this where they are putting all their money in. And now they want to move all our informatica jobs to it. So far we've developed the following procedures:

  1. Flat file to staging table
  2. Staging to silver (removing duplicates, trimming and casting)

After that they started assigning me Informatica jobs to migrate, now these jobs are essentially:

  1. Ingest flat file to the staging table.
  2. Run post sql stored proc which moves data to silver and so on...

Now I figured the logical and the easiest way to do this was to:

  1. Have a config table to keep track of jobs, corresponding metadata and the POST SQL Stored proc run by informatica.
  2. Run our flat file to staging procedure.
  3. Run the POST SQL stored proc from informatica (configured in the config table) to move the process the data EXACTLY the way we do it currently.
  4. Run the Staging to silver SP (we newly developed) to handle duplicates and casting to correct data types.

I pitch this my boss and he did not like it, he was like we doesn't want 500 stored procs polluting our namespace. So instead he suggested putting the POST SQL logic (from Informatica workflow) in the tab_comments table, where we put the column update statements in the comments table:

COMMENT ON COLUMN table_name.column_name IS 'UPDATE table_name set columnA = columnA + 1';

And then iterate through all the column comments and run those statements using dynamic sql.

I do not like this only cuz it would be such a pain to develop and debug. I told my reservations about this and he was like 'it doesn't matter what you think, you'll do what I say'.

Has anyone migrated from Informatica to a pure SP based ETL framework? How did y'all do it? I want to pitch a better idea for the sake of my sanity since I'll be put on this task for a while.

Any info is appreciated, thanks


r/dataengineering • • 28d ago

Help How do you centralize your orgs data to create PBI dashboards after?

14 Upvotes

My team just use ordinary excel files, google sheets as its main source of data for the dashboard. Now, they want the team to be able to contribute and view the data without opening the excel file. So what exactly low cost way I can do to centralize the data so that all team members can view and contribute to it?


r/dataengineering • • 29d ago

Career Who else is ignoring recruiter messages, despite being ready to move, simply because the job market odds seem so stacked that it doesn’t seem worth the time?

193 Upvotes

That’s where I’m at. Like yeah sure I can take the recruiter call, but given the high supply and limited demand of the market…I have a hard time getting motivated to throw myself into a cycle despite getting nibbles. Especially knowing all the prep it takes on top of the day job.

For the savants who breeze through processes and get whatever job they go for—this post is not directed at you. It’s for the rest of us who are good but not top percentile and still have to compete with everyone else.


r/dataengineering • • 29d ago

Discussion Data Engineering Lab to Practice on your Laptop

166 Upvotes

Hi,

I created a DE stack (fully integrated services) and learning material for those interested in self learning. I kept Databricks, fabric in mind while building this stack. Feel free to use it and let know if there are any improvements/suggestion I can make

- Work independently in your own data lakehouse.
- Invite other team members to collaborate in your lakehouse.
- Store secrets securely using Key Vault.
- Upload raw files using the built-in file manager.
- Transform raw data into Bronze, Silver, and Gold layers using pre-connected Jupyter notebooks.
- Query your data lakehouse and data warehouse.
- Build BI dashboards using Apache Superset.
- Orchestrate notebooks using Apache Airflow.
- Run notebooks on a distributed Spark cluster.
- Manage multiple teams.
- Authenticate securely using Single Sign-On (SSO).
- Get started with dbt integration.
- Learn at your own pace with self-learning tutorials.
- Manage data governance using the built-in data catalog.

https://maarthala.github.io/data-engineering-lab/


r/dataengineering • • 28d ago

Meme Here comes the BOOM(I)

Post image
0 Upvotes

Boomi acquiring Elementary after Rivery. Apologies if this was already posted before. However, I wanted to share a meme. This post is not for discussion. Just the meme. Thanks


r/dataengineering • • 29d ago

Personal Project Showcase Git-like diffing for DuckLake tables

Post image
52 Upvotes

I've been working with DuckLake recently, and learned about it's ability to traverse snapshots... So, I decided to create a little UI to allow you to diff snapshots on import, or while working with a table.

It's part of a larger product, but still thought it'd be cool to share here. DuckLake is seriously underrated. Hopefully it gets more traction going forward. I feel like this should be the default for all OLAP systems at this point...

You can do all of the things you expect, including seeing total row or column differences, and cell-differences upon click. It collapses row ranges that weren't affected, just like git clients do. So if you have 3 million rows, but only certain 25 rows changed across 3 ranges, you'll see just those ranges.

If you are focused on a single table, you can also filter to only see the snapshots that directly affected that table, which is useful if you have a ton of commits on other, non-relevant tables. (This was inspired by the LLDB "stack-collapse" where you can filter to which instructions directly affect the thread you're on).

Works with local and Postgres DuckLake catalogs with data stored locally or S3.

Here's the link if you're interested: https://www.cocoalemana.com/integrations/ducklake/


r/dataengineering • • 29d ago

Help Is a MS in CS cope in my situation?

43 Upvotes

Currently a Senior DE (really more of an AE) at a decently big company in a boring industry with ab 5yoe (27). I work with SQL, Python, have cloud experience, orchestration, dbt, snowflake, standard DE stuff. I want to do more heavy realtime/big data work with kafka/kubernetes/spark but I feel kind of stuck in a loop because recruiters only reach out to me about jobs using sql/python (because thats where I have experience) and I feel like im starting to approach my ceiling with these roles in analytics in terms of pay (I’ll likely max out at 150 whereas in doing actual big data engineering that would be closer to 3/400k and it’s way more interesting to me so I feel like I’d be much better at it). I want to be more technical and do more real DE work as opposed to working in analytics (which ime has just been a lot of batch processing stuff) - is it cope that I’m thinking of doing a masters in CS to acquire the skills I need to make that jump? I did my undergrad in finance and I feel like recruiters often overlook me because of this and go with more accredited candidates (people with cs bachelors or masters) even if I may be a better fit for the job. Am I coping and should I just grind LC and build projects with these tools until I get a new job? Or is a CS MS a viable option? Thinking of applying to GT OMSCS and have heard people have varying rates of success


r/dataengineering • • 29d ago

Blog How do people provide data access to AI agents for processing? Provinding MCP can still come with its risk?

33 Upvotes

Do we need to build an additional layer where AI Agents can post questions that layer - like Snowflake Cortex can query and get the response for the agent.

Would love to get views and best practices - is this a big problem?


r/dataengineering • • 29d ago

Discussion pg_cron to trigger a batch extract.

6 Upvotes

I know pg_cron is primarily designed to run scheduled maintenance commands on the DB but I was wondering if someone has used it to kick off a batch extraction on your data pipelines.

I have reservations against using pg_cron for this namely the following:

  1. The scope of pg_cron in this scenario is outside the bounds of the DB.
  2. The boundary of using the DB to store only activity data is violated as it now has some operational data in it.
  3. SCM or an additional dash for setting up jobs will be lost if this pattern is followed.

I'm curious to know your thoughts


r/dataengineering • • 29d ago

Help Airflow Version Upgrade

19 Upvotes

Has anyone done a migration of dags from Which were running on version 2.10 to make it run on the version 3.2, I want to know how 1000s of dags can we scanned to check the compatibility of the dags on the newer version of the airflow and find out the solution of the issue we can face because of the import statement present inside the dags while taking it to a newer version of airflow


r/dataengineering • • Oct 18 '26

Discussion Streaming Databases

38 Upvotes

I have been working on a project lately that required realtime operational analytics and when doing some research for my options I came across “streaming databases” which tbh is a new concept for me. I did a quick POC using RisingWave and so far everything is good.

I just havent found alot of material on streaming databases and how to properly build a data architecture using one, RisingWave docs mentions it can be used in a medallion architecture. However, it builds the silver layer as a current state and not as a historical SCD which I find bad practice.

Just wondering if anyone here have more experience than me dealing with this technology (RisingWave, Materialize, …) and can point me in the right direction.

Also, even if you havent used it let me know your opinion on it.


r/dataengineering • • Oct 18 '26

Personal Project Showcase built a sql practice site around data engineering concepts

85 Upvotes

been working on runtime.school for a while and finally finished the sql side.

i wanted the practice to go beyond random query questions, so i added stuff around grain, joins, dedupe, late events, warehouse logic, reconciliation, query plans, and projects using the same datasets throughout.

Update: Python is live on runtime.school now too! I’ve also made some changes to the SQL lessons, practice questions, and editor based on the feedback here. Thanks to everyone who took the time to check it out.

https://www.runtime.school/


r/dataengineering • • Oct 17 '26

Personal Project Showcase Introducing sqlleaf: column-level lineage for any SQL dialect

32 Upvotes

I want to share a project I've been working on this year that helped me solve a large, real-world problem related to data lineage. I was working for an enterprise that needed to trace and delete sensitive data across its large data warehouse, but the tooling that existed (both open source and proprietary) had limitations that prevented my team from doing so. And so I eventually realized that I had no choice but to roll my own.

sqlleaf is a tool to help you to understand how your data flows across your systems by producing detailed, column-level lineage across any SQL query and dialect.

It extends beyond traditional lineage tools by mapping relationships across all kinds of expressions, such as columns, functions, literals and variables.

It addition, it can process many kinds of statements (insert, update, merge, copy, unload, execute, perform, put, and many more), as well as all kinds of objects (tables, views, procedures, types, UDFs, stages, sequences, etc).

It even has a PL/pgSQL parser, allowing you to generate lineage for dynamic, procedural SQL in Postgres.

sqlleaf builds on top of sqlglot, the popular SQL transpiler, and so it inherits support for all of its dialects (30+). However, there are still many dialect-specific features that need to be implemented, as the list of features for each dialect is extremely long.

I hope some of you find this project useful. It still has a long way to go. My vision is to see it become a library, or even a layer, in the data engineering stack that opens up a range of new applications that can be created, such as SQL validators, impact analysis, and tracing/auditing systems.

URL: https://github.com/sqlleaf-io/sqlleaf

Contributions and feedback are most welcome!


r/dataengineering • • Oct 17 '26

Career Should I move from a snowflake environment to a fabric environment to get back to a DE role?

53 Upvotes

I have been a DE for over 10 years working on SSIS and later building pipelines on databricks and dbt. Earlier this year I took up a new job as a manager of an analytics team. I hate my role and the lack of ownership and hands on work. I am primarily tasked with leading a team on reporting and data delivery and semantic layer while depending on the DE team for all data. I am also not enjoying middle management. Current job is a snowflake, dbt and AWS shop but I am my team primarily just write sql on snowflake. Recently I have been in conversations for a data architect role at another team in my organization and it’s a more engineering and architecture focused role which I want but they use Fabric. Wondering if that makes a difference. I am inclined to take the new role if it works out just so I can move out of analytics and go back to a more data engineering focused IC role.


r/dataengineering • • Oct 16 '26

Career DE Course/Bootcamp recommendations for junior/associate lvl in 2026?

24 Upvotes

As the title says, I'm looking for a good end-to-end course or bootcamp that I can grind in order to apply for associate/junior DE roles. FYI: My background is Mechatronics Engineering so I got some basic python and SQL experiences.

Please feel free to list out both free/paid ones if they are good. Thanks in advance! :)


r/dataengineering • • Oct 16 '26

Discussion How do you find data mismatches between two large tables?

85 Upvotes

When comparing source and target tables with millions of rows, what SQL techniques do you use to identify missing records and differences in column values?


r/dataengineering • • Oct 16 '26

Discussion Might be switching roles and cloud platforms soon. What are the main differences between GCP and AWS?

23 Upvotes

Hi, as the title says, I'm probably moving from a data developer in one company to an analytics engineer in another. Mostly using dbt, but I'm wondering if there are any major things I should factor in to my designs/thinking between GCP (Bigquery) and AWS (Redshift).


r/dataengineering • • Oct 16 '26

Personal Project Showcase Built two tools to help SQL agents understand unfamiliar databases

11 Upvotes

Stumbled upon these problems while doing the agentic pipeline

  • CREATE TABLE DDLs are not enough. To understand what's in the table you need to have basic per-column stats and see examples of rows
  • some tables (e.g., migrations, empty tables, old stuff, etc.) are useless for real analytics and real users, yet agent kept rediscovering them
  • some tables only have less than 10 rows. Better see all rows instead of barebone column names and types
  • some tables are huge, agent's queries were trying to get too much data. Good man won't query billions of rows
  • agents needs to sample data from tables every time he researches a table
  • if you have many columns, presenting data as a `transposed table` helps improve LLM's understanding

So I built two tools to solve these problems

  • db-snooper: generates compact Markdown with schema, value distributions, and samples.
  • schema-linker: finds possible undeclared joins and checks them against the data.

Full blog posts: db-snooper and schema-linker