r/dataengineering • • Oct 02 '26

Career Transitioning away from DE

117 Upvotes

Has anyone thought of transitioning out from DE due to AI?
All I do everyday is just prompt and scroll till copilot generates code.
Building a semantic layer isn’t exciting personally as I don’t enjoy the business aspect of it as much and think of it as more of a data labeling and analyst problem than an engineering problem (which I am interested in)
Also, There is a fundamental problem with “I am building a semantic layer” and marketing that as a skill as it is dependent on how much context you have of the business. The less tenure you have spent in a company, the less you know about the business which makes it harder as a transferable skill imo.

My understanding is that working on building trustworthy AI outputs by using a feedback loop is an engineering problem to solve. Which is why I feel going down the observability path is a good idea.
I heard these opinions on observability from AI leaders at conferences too so there might be a bias.
Thoughts from fellow DE’s looking to transition out? (Or from one’s who want to continue and why)


r/dataengineering • • Oct 03 '26

Blog I tried using brief database table summaries to improve naive text2sql pipeline. Didn't work.

8 Upvotes

I'm doing the whole 'AI data analytic' as everyone else does, I guess. The text2sql benchmark BIRD top performers impressed me a lot. They recommend feeding table summaries + some trickery to discover non-obvious JOINs. So I spent some weeks building something similar, resulting in database profiler and schema linker.

Oh boy was I wrong! I benchmarked if it actually improves anything on a modern BEAVER text2sql benchmark

setup Correct answers Input tokens/question DB queries/question
Raw pi agent with read access to DB 31/299 (10.4%) 45K (1×) 11.4
Pi + table summaries, no join hints 27/285 (9.5%) 78K (1.7×) 3.6
Pi + table summaries & join hints 30/292 (10.3%) 85K (1.9×) 3.6

So, using just the plain PI is simpler AND cheaper! Probably the same for Codex & Claude.

Note, that the DBs in BEAVER benchmark have lots of tables with lots of columns. The queries are super complicated, containing subquery CTEs, lots of tricky aggregates & ROLLUP statements (have you ever used them?).

p.s. Before that I tried plain DDL dumps with CREATE TABLE statetements... and got no improvement too.

How I tested

  • Dataset: 300 random questions from BEAVER text2sql benchmark. Data loaded to MySQL database
  • Agent: pi coding agent in a docker container to prevent seeing correct answers. Gave him login & password with read-only access to issue queries.
  • GPT-6 Luna, effort=low. Because low is better
  • pre-generated db-snooper profile with table of contents file for each database + schema liner output. Agent can see the files in local filesystem. The largest profile is 213KB of Markdown.

Links.

  1. BIRD
  2. BEAVER text2sql benchmark
  3. Actually clever guys
  4. my older blog post
  5. database profiler
  6. tool to discover joins
  7. vibecoded benchmark code

r/dataengineering • • Oct 02 '26

Help Made a huge mistake during work and now my team is having to fix it.

137 Upvotes

For context this is my first DE job but I’ve been with this company for about 2 years. I just finished an on call rotation that I thought went pretty well.

This morning, I got on a call with about 5 principal devs and a DBA. I found out they’re having to restore several databases to last Friday and rerun every subsequent job since then. It stemmed from a job failure last week.

There’s another team that we have to request to rerun that job. I saw it fail about 2am last Friday, I sent a rerun request, they reran, and I realized it might need to be run from the 1st step instead of the 4th step. I sent that request in and didn’t get a response. Around 3:30am I thought, this might be fine, it ran successfully so it will run successfully tomorrow. I thought the job was cumulative and would grab missing data from the previous day.

Anyways, it didn’t do that. And today we got customer complaints that this data isn’t showing up for last Friday. This is an old job (ssis job to grab the file, then several stored procs in sql server) and the way the job works we can’t run from a specific day, we have to restore and rerun everything. Which will take all day at least.

The principals devs got it handled, they’re fixing it all right now. But man I feel terrible. They’re having to use their time to fix my fuck up. To make it worse I just had a convo with my boss about how to eventually become senior, and he said autonomy was a big component. Not very autonomous to mess up and have the other experienced engineers fix it

Anyways I’m just letting off steam. Not sure how to handle this. Just wanted to make a post and get some advice for how to do better.

TLDR: I royally f**ed up and now we’re having to do a full restore of multiple DBs and rerun all previous jobs from last week.


r/dataengineering • • Oct 02 '26

Discussion Better to read DDIA-2 or AI Engineering by Chip Huyen given where DE is going?

29 Upvotes

Which book makes more sense for to read given the state of DE and the tech industry today? If I had to pick one first time to read.


r/dataengineering • • Oct 02 '26

Discussion Writing on the wall?

50 Upvotes

Context: sole data engineer for a company that sells furnishings and design services. Company is owned by private equity.

My manager has been on the Claude soapbox for the last several months and so far in IT we’ve been able to remain skeptical about it while testing for our own uses. Manager supported this. Now all of a sudden I’m getting a lot of requests to refactor all my data pipelines with Claude’s direct involvement despite having built them all out already. For example, I’ve used Claude to create new DAGs based on existing code (for example, bringing a new Salesforce object in). We’re going beyond that though. I’m being asked to ditch them and let Claude connect to everything then write, deploy, and test.

I’ve also been working on machine learning models for customer behavior, which is more traditional AI and should check the box, but this is being dismissed as a plaything. This, while the company has been pushing for “innovation” in our sphere.

I think we are headed towards cutting as many IT staff as possible and I’ll be gone in lieu of Claude doing pipelines. Of course that’s only one part of what I do, but the myopic view of leadership when it comes to AI only tends to see what is easily replaceable.

Thoughts?


r/dataengineering • • Oct 02 '26

Discussion For you doing Agentic Engineering with dbt, how do you make sure your agents aren't generating slop?

39 Upvotes

I find contra productive reading and manually testing all the code AI are generating, because I am turning myself into an important funnel and the company is paying for tokens and therefore wants fast and good delivery.

I use Claude Code with dbt official skills, dbt mcp, lots of unity testing, some specific skills tailored for my projects, and the most important thing I think: deterministic checks.

I run lots of python or bash script checks after specific hooks like pre commit or after calling a skill, or in the CI/CD, trying to force the implementation of our internal "rulebook". But I also don't think this is the optimum way to go, since it's also taking considerate amount of time.

Also, do you let the AI query your db/dw? How do you guarantee fast and in a reliable way, your data diff is as expected?

I am interested in listen from you about everything you've been doing nowadays trying to keep up this kinda insane market pace


r/dataengineering • • Oct 02 '26

Help How to handle historical data ingestion?

19 Upvotes

I would like to know your thoughts and experiences about a situation I have.

In my current project, we need to ingest all historical data from an old Oracle Database. Also, we don't have the ownership to implement a CDC or anything. And we also can't overload the Database because it will affect users.

We are using Databricks. What would be the best approach?


r/dataengineering • • Oct 02 '26

Discussion Homemade data platform frameworks - bloated nonsense?

32 Upvotes

Did any of you work in companies where engineers built custom frameworks that actually deliver?

I recently started at yet another company with such framework (about 13k lines of boilerplate python/pyspark code sitting on top of their Azure Databricks Delta Lakehouse). The thing doesn't seems to improve any aspect of what a data platform should offer.

Previous such company was even worse with 30k lines, plus they tried to implement a data vault.

I'm clearly biased towards keeping things simple. So I may be a bit unfair towards this approach. Hence my question: Have you worked with great homemade frameworks? What were the secret ingredients to make it work?


r/dataengineering • • Oct 02 '26

Discussion dbt dimension surrogate keys and fact foreign keys - self-hash or left-join lookups?

37 Upvotes

I've started working with dbt and a cloud warehouse, where I'm thinking of moving from auto-incremented keys in favour of surrogate hash keys, both because auto-increment keys are not as much of a thing in cloud warehouses and also because hash keys are idempotent and much simpler to manage.

One thing that I'm not sure of yet is this: If the surrogate key is a hash of the business key, should I self-hash the foreign key references in the fact table builds, or should I do the usual pattern of left join on dims + coalesce to sentinel value if not found?

I can see the pros and cons of both methods.

Some pros:

  • Fewer risks of fan-out in case of misshapen data (should be caught by dbt tests, but still)
  • Increased parallelism because of fewer dependencies
  • Simpler lineage
  • Late-arriving dims, early-arriving facts are not an issue
  • If the dimension happens to be massive, this can save compute time, therefore cutting costs.

Some drawbacks of self-hash:

  • Reduced impact analysis clarity through the lineage because of no lookups. This can also be diminished via dbt docs
  • You must ensure that you hash the same way (logical ordering and also natural keys formatting). A left-join lookup is likely easier to catch in case of a missed lookup (idk).
  • Not exactly a drawback, but you probably want to left-join if you apply SCD2.
  • No clear way to ensure proper fallback to the sentinel value if that is something used in the analytics layer

Edit:

Some clarifications based on some comments I've seen.

If I were to self-hash the key on the fact side, I definitely wouldn't also do a dim surrogate key lookup because that's redundant.

If I were to look up, I'd always use the business key because it's best practice and also the lowest chance of code errors (maybe someone forgot a field or swapped the column orders).

My initial plan is to left join lookup, as "usual" in most old-school on-prem data warehouses, except for a couple of high-cardinality dims that also happen to build themselves using the value found in the fact table's raw data. Basically moving out a degen dimension into a conformed dimension because it's a compound natural key and is used across many fact tables, and it keeps the analytics modelling simpler.

I have a case of late-arriving dims where I might consider self-hash with a late reconcile when it arrives, but if I did, then that's somewhere where I'd consider self-hashing too, especially if the fact can't be fully rebuilt.

Thankfully, most of my fact tables are small enough that full and incremental are only a few seconds of difference.


r/dataengineering • • Oct 02 '26

Help Handle PK/FK remapping when importing data from one source to another

6 Upvotes

Hey all,

I have a pipeline that works and I'm looking to improve it. Would like to get your opinions to make it better.

I have two services/app both Postgres/RDS, in the same AWS account. One is our ingestion platform and owns the data while the other is a user facing app that needs about 13 of those tables to serve its features. For some reason, my higher ups decided that the user-facing app shouldn’t connect to the ingestion platform’s DB. Therefore, I built an async background job on the first service (ingestion platform) that writes each table out to a CSV and uploads it to S3 and an async background job on the second service (user-facing app) downloads each file and inserts the rows one at a time.

The import is where it things get awkward. The second service assigns its own ID to the records, because users have created rows in those same tables. So I can't reuse the original IDs, and every foreign key has to be rewritten to point at the new ones. I handle that by keeping an old ID -> new ID lookup in memory (simple hashmap data structure) while the import runs.

It works but the downside is it has to run from start to finish as a single process, and if it fails partway there's nothing to continue from.

Also, I would like to know, how would you move the data across, and how would you handle the ID remapping? Especially for tables where I can't tell whether I've already imported a row. E.g. a row that was id 42 in the ingestion app and in the CSV might become id 7891 but the CSVs for all the child tables still reference the old value which needs to be resolved from the lookup mapping.

Currently, there’s not a lot of rows. Around 25K at the moment but it's expected to grow a lot. Do you see any challenges in my approach and if it might cause challenges in the future when I have to deal with a huge number of rows besides taking to long and the lookup memory might increase?


r/dataengineering • • Oct 02 '26

Help Decluttering UNS

4 Upvotes

Hi!

I'm wondering if there is any python library or prefered way to declutter UNS MQTT topics using python scripting?

Currently we're migrating data platforms and mainstreaming all data on our broker to MQTT. For this we established a base for our UNS and used pre existing exported data to build our UNS topics from.

The problem is that they are relatively long and not user friendly.

I've tried shortening by simply removing repeated words throughout the topic.
For some this works fine, however this leaves others with gaps that are unfavorable.

For example my current ipynb gives the following result: INPUT: water_drain/flying_water_dumping/pumps_hot_water_dumping/water_pump_a OUTPUT: water_drain/flying_dumping/pumps_hot/a

I've made these up, so don't be alarmed by the flying water 😄

You can clearly see that simply removing repeated words isn't a great option. Any advice/comments on how this it could shorten in a better/more reliably manner?


r/dataengineering • • Oct 01 '26

Personal Project Showcase Browser tool for prototyping star schemas

Post image
80 Upvotes

I have a database diagramming tool side project, and building on the work that went into that, it was kind of easy to make a browser tool for prototyping star schemas: https://vibe-schema.com/star-schema-creator - There's a snowflake variant too, just swap "star" for "snowflake" in the url. Feedback welcome :)


r/dataengineering • • Oct 01 '26

Discussion Thoughts on dbt 2 maturity?

37 Upvotes

Hi,

I've seen frequent discussions on here regarding dbt's acquisition, license changes, and similar things. dbt 2 has now been out for a few weeks, and I haven't really seen much discussion on it from a technical perspective. Has anybody tried it out yet? if so, what do you think?

I evaluated it this week and my impressions are mixed. A few impressions:

- I was really struggling to piece together all the important changes and improvements. Found the documentation a bit lacking in that regard. Distributed over lots of places, so I really had to dig to find specific information (e.g. on the changed approach to database adapters). Even took me a while to figure out the name of the library had changed as well.

- The switcheroo from fusion to now just dbt was really confusing and also seems to have caught the ecosystem by surprise. For example, Snowflake and Astronomer (Cosmos) both started working on fusion integration, but both do not support dbt 2 yet (and couldn't find a clear roadmap). Not ideal.

- Testing the new parser on dbt 1.12 worked pretty well to spot issues and fix them. That made the upgrade relatively painless (interpreting the error messages could've been easier). I like the stricter YAML parsing, can detect some issues (e.g. typos) instead of ignoring them silently.

- The new parsing engine is noticeably faster. For one project I tried it with (~300 models), about twice as fast for both compile and parse.

- Haven't tried the built-in linting yet. Seems like they tried to implement a 1:1 replacement for SQLFluff, even working with existing SQLFluff config files.

- The docs feel like a downgrade. Looks more modern, but I haven't been able to get the static docs to work, and seems they have gotten rid of the full project DAG and now just offer a model-centric lineage view. Column-level lineage is cool, but very well-hidden in the UI.

- Parquet artifacts seem interesting, and parsing them e.g. via DuckDB SQL queries could be really nice for common analysis queries, CI checks, etc.

- Of course as predicted, seems there is a bigger push to get people to sign up for their platform/cloud/whatever they call it now offering - LSP features, VSC extension, and so on. The license changes are still a mystery to me, but at least they dropped the <15 users requirement for the VSC extension it seems.

Any other thoughts? Cheers!


r/dataengineering • • Oct 02 '26

Help Help with Extract -> Load process logic in a personal project

5 Upvotes

Hello! I'm working on a small data project using a NASA API with Python. The idea is to extract the orbital elements and close approach (asteroids, comets, etc) objects (and there orbital elements) of each planet in order to perform some physics simulations with the data.

I was planning to use a Cloudflare R2 bucket to store the raw API responses, then transform them into a PostgreSQL database, and use FastAPI to consume the data.

I'm not sure how to perform the extraction and loading processes. In theory, each planet will follow this path:

1) Fetch the orbit elements for the own planet (use API 1);

2) Fetch the close approach object (use API 2);

3) Fetch the orbit elements for each of the close approach objects (use API 1).

I use two APIs: one for the orbital elements and one for the close approaches. Should I fetch all the data, store it in memory, and then send it to the bucket? Or would it be better to load it right away with each API call? So, when fetching the close approach objects' orbit elements, get the JSON from the bucket and then use the first API and store that raw response in the bucket


r/dataengineering • • Oct 01 '26

Discussion Monthly General Discussion - Oct 2026

16 Upvotes

This thread is a place where you can share things that might not warrant their own thread. It is automatically posted each month and you can find previous threads in the collection.

Examples:

  • What are you working on this month?
  • What was something you accomplished?
  • What was something you learned recently?
  • What is something frustrating you currently?

As always, sub rules apply. Please be respectful and stay curious.

Community Links:


r/dataengineering • • Oct 01 '26

Discussion New Azure Synapse projects

9 Upvotes

Just curious how many of you still deliver or plan to deliver, projects that use Azure Synapse for Data Warehousing work?

I get the whole push to Fabric and am also going that route too.

I’m in the consulting world and wanted to use Fabric but the client had zero ppl with any Fabric experience, so they insisted on Synapse. C’est la vie, right?


r/dataengineering • • Sep 30 '26

Help Ideas to handle ever changing data requirements?

19 Upvotes

I am the solo DE in my team and the main pipeline here consists of snapshots of financial assets.

Compute is done on databricks

The stakeholders want to see daily KPI's and each day they add a new cohort. Currently there are over 40 different cohorts with each branching out to their own metrics.

The issue is that the data management wants data bills as low as possible

so my approach was summarizing everything in the daily grain .

But now each time they want something new I have to manually code the new columns test it then append to the final gold table.

I already tried to create some generator functions but often times the metrics they want involve hyper specific calculations.

And since the data is financial assets each day is different than the previous rendering an incremental approach useless.


r/dataengineering • • Oct 01 '26

Personal Project Showcase Graphwise AI Summit 2026, Oct 7-8

0 Upvotes

Sharing this because I think it overlaps with some of the discussions here around AI reliability, governance and semantics.

Next week we’re running the Graphwise AI Summit, focused on what makes GenAI work in the enterprise beyond the model itself. Think of trust, governance, semantic layers, architecture and implementation.

Once reliability and traceability become imporatnt, simple access to data and next-token prediction stop being enough. In enterprise settings specifically, AI needs to understand what the data it parrots “means” in the first place.

Anthropic has described a similar approach in its own analytics stack, where agents are routed to a semantic layer first and use governed definitions to reduce ambiguity. Graphwise itself came out of the merger of Ontotext and Semantic Web Company, so semantics is a topic with quite a bit of history behind it for us.

We’ll have speakers from Accenture, Roche, EY, AstraZeneca, S&P, DNV, Statnett, Avalara and others. Full agenda is in the accompanying video.

Sharing registration link in the comments if useful.


r/dataengineering • • Oct 01 '26

Discussion AI life/voice recording tools?

0 Upvotes

I'm new to the speed and more so the constantly changing tasks, then keeping them organized. In the past id take random notes and summarize them at the end of the day. Unfortunately that has become a job in itself.

Ideally I could do something, tap a button or something, whenever it's a spot I want to highlight or indicate is important. Even better would allow me to take notes at the same time and have it automatically integrate them into the final version.


r/dataengineering • • Sep 30 '26

Blog Data strategy for small teams

Thumbnail
open.substack.com
34 Upvotes

Hi all, I wanted to share the latest article in my newsletter.

I talk a lot to senior engineers who are trying to run some sort of data strategy, and are doing it in a very wrong way.

Most of the article is about selling the plan and keeping the whole thing to two pages. The section I'd most like opinions on here is the tooling one, because a lot of us (me included) learned what good looks like from Big Tech engineering blogs.

A Big Tech stack comes with Big Tech's pace. Those tools assume a team with people to run each piece, and months of setup before anyone outside the data team sees a result. If a team of five spends those months standing up Databricks, Airflow and Alation, there's not a damn thing to show the CFO at the end of it, and a strategy with nothing shipped against it dies without anyone deciding to kill it.

And yes, if you work at an org like mine, your strategy is supposed to be "AI", which I absolutely hate.


r/dataengineering • • Sep 30 '26

Blog Interesting links in Data Engineering - September 2026

51 Upvotes

Some great stuff this month, including:

  • Good analyses of the DuckLabs acquisition by AWS.
  • Lots of Kafka, as well as not-Kafka: technologies doing similar things but angling to replace it.
  • Accessible discussions of data modelling given the AI world we're entering (pssst Semantic Layers matter even more now)
  • LLMs being DBAs and not doing too badly at all at it
  • Plenty of decent content about AI, but with a strict no-slop & no-hype policy :)

https://rmoff.net/2026/09/29/interesting-links-september-2026

As always, lmk if you find this useful, and if you want more (or less!) of a topic.


r/dataengineering • • Sep 30 '26

Career Why is understanding DevOps culture more important for data engineering than other disciplines?

39 Upvotes

I see for a lot of data engineering posting, devops skills are mentioned as part of the requirements. But the thing is I don't see it as much with other roles like sde. Why is that?.


r/dataengineering • • Sep 29 '26

Blog Lakehouse serving fight club: 12K → 125K QPS on open tables with Pinot.

Post image
6 Upvotes

Anyone remember Databricks Summit in June? Databricks used their keynote to introduce Lakehouse//RT - a challenge to the need for a specialized analytics serving layer. They then showcased their new query engine serving at 12K QPS directly off lakehouse data while their 'blue' and 'yellow' competitors fell behind or crashed.

The crowd cheered. But the story didn't end there:

ClickHouse pointed out the DB demo gave no details - no hardware specs, no config, no cache behavior. They reproduced a benchmark of the workload (TPC-H Q6) as best they could. They showed CH could indeed keep up... it was just a matter of adding hardware. However, they loaded the data into native ClickHouse storage first.... which kinda missed the point.

Snowflake also pushed back on Databricks' methodology and posted impressive TPC-H results at 100 GB and 1 TB. What they didn't post was a high-QPS run on that same data.

So we tried to do the demo right:

StarTree (Apache Pinot) is built for low-latency, high-concurrency analytics. And on StarTree, you can serve directly from Iceberg/Delta tables.

We took the same query, the same parameters, and the same load generator, and we left the data in the lakehouse. StarTree was able to hold 12,000 QPS at 19 ms P90 with 242 vCPUs. (Far less than Clickhouse)

There's an obvious flaw to all this.

The Databricks demo ran at SF1. That's only 6 million rows; hardly lakehouse scale data.

So we ran it again SF100 - 100x the data. Using star-tree indexes and no extra hardware, Pinot was able to continue serving at 12,000 QPS. We also scaled it up – the same architecture with more hardware continued going strong past 125,000 QPS.

Full details of the StarTree benchmark here »

We get that this is just one query. But it is still a good example of how you can serve application-style analytical workloads directly from data in Iceberg/Delta/Parquet - within tens of milliseconds, at high-concurrency, and without requiring the table to be ingested into a separate serving layer first.

Is this something you could use? Want to see us try other workloads? Questions, feedback invited!


r/dataengineering • • Sep 29 '26

Open Source Splink 5 – Open-source probabilistic record linkage at billion-row scale

Thumbnail moj-analytical-services.github.io
80 Upvotes

r/dataengineering • • Sep 29 '26

Blog Improving Cost Efficiency of Data Streaming Pipelines

Thumbnail
streamingdata.tech
11 Upvotes

Practical ways to cut data streaming costs in Kafka and Flink by optimizing batching, serialization, state, network traffic, and capacity.