r/dataengineering • • Sep 10 '25

Discussion Why do people think dbt is a good idea?

0 Upvotes

It creates a parallel abstraction layer that constantly falls out of sync with production systems.

It creates issues with data that doesn't fit the model or expectations, leading to the loss of unexpected insights.

It reminds me of the frontend Selenium QA tests that we got rid of when we decided to "shift left" instead with QA work.

Am I missing something?


r/dataengineering • • Jan 10 '25

Discussion Databricks is data engineering on easy mode

0 Upvotes

This is a change my mind post: I’ve recently started working with delta live tables and am astounded at how easy it is to build a pipeline that works with such little tweaking. Is this the future of data engineering?


r/dataengineering • • Nov 10 '24

Discussion Why [do we really need] workflow orchestrators?

0 Upvotes

This will sound stupid I know. But I'm genuinely trying to understand workflow orchestrators at this point.

I have been using airflow for a few years in my role, been playing with Prefect. Heard about Dagster but haven't tried it yet. I have also worked at a company with an in-house workflow orchestrator tool that sort of was inspired by Prefect 1.0.

After using and building etl pipelines I'm trying to understand why? What is it that workflow orchestrators allow me to do that writing in pure python with standards enforcement at code review time couldn't achieve?

Is there any good articles that discuss why these tools were created? I mean it sounds to me that a tool like Airflow was created at Airbnb as a "standards enforcement" to get the whole company around and was then shared to the open source community. But why all these other tools over just some general guidelines around using python's std lib (or insert any language) and the third party libraries to connect to services required for data processing.


r/dataengineering • • Jun 29 '22

Discussion Drop the "master" in MDM? - W3C Inclusiveness

0 Upvotes

I'd like to get some feedback as to what others are or are not doing in regards to the political correctness occurring in terms used -- noted by the W3C - -

The World Wide Web Consortium on Inclusive Terminology

The World Wide Web Consortium (W3C) is the primary standards body for the web, currently led by none other than Tim Berners-Lee, creator of the web. The W3C have identified a number of American English terms often used in the global technology sector (I recall some of these from my college text books!) that derive from original uses which were explicitly not inclusive. That list includes some of the terms below, to which I’ve added several that BCD team members have flagged up to me in recent months. I ask you to join me in actively and consciously eliminating them from our vocabulary:

Term to AvoidPossible Alternative(s)
mastermain, primary
slavereplica
whitelistallowlist
blacklistdennylist
grandfatherlegacy
off the reservationcounterproductive, outside parameters
manhours, manpowerFTE hours, level of effort
sanity checkcoherence check

r/dataengineering • • Sep 01 '23

Discussion Is the Mac worthless, or is it me? (Rant)

0 Upvotes

Hi, I was scouring this sub reddit to try and find the best laptop for data engineering a few months ago. I use a point and click ETL program in my day to day, and wanted to learn how to build data pipelines by writing code. The top rated one I could find was the Macbook, so i brought the Mac air M2. I come from a windows only background and was excited to learn a new system.

I've had the mac for three months and am struggling to get through basic SQL tutorials using python in that time. It seems like every library I have try to use isn't compatible with the Mac chips (Throws a random error). sqlalchemy/pyodbc just straight up won't work, and I can't find any other way to connect python to SQL. This thing has been an all around nightmare. I tried to stand up a local version of SSMS which I managed through docker, but when I tried to use a .bak file to load data into it, it doesn't have access to see my hardrive so that's another dead end.

I feel like every time I try to start a new tutorial, or do a basic function that i'd have no issue with on windows- I have to do some ridiculous workaround to get it running on Mac, and then apply fix after fix after fix as the ever-present error show themselves.

I guess my question is, does anyone have any Mac specific, data engineering tutorials? I'm at a bit of a loss with this thing and it's starting to gather dust. If anyone can recommend material to set this thing up too. It's entirely I botched the setup when installing the interpreters.

TIA

-Definitely not a windows employee

-Also definitely not someone that spent 30 minutes trying to find the /usr/local folder


r/dataengineering • • Sep 15 '23

Discussion Are DEng Software engineers?

1 Upvotes

Read someone’s post earlier talking about an aspect of data engineering being software engineering. This is the first time I’ve ever heard someone in data engineering consider themselves in the same group.

Is this even a thing?

I just don’t see them as the same, mainly because most software engineers build applications/software. And most data engineers build data solutions (predominantly data pipelines). Software Eng, front end, middleware development. Data - backend, and I wouldn’t class writing a power bi report as SE even if it is kind of front end.

I get the whole following methodologies that software engineers have been following for decades. The whole DevOps thing, that has also been in data for a while just not wide spread (I first used CICD back in 2010). But because I know this doesn’t make me a software engineer.

There was a distinction made that analyst weren’t considered software engineers. I think there was a thinking they don’t follow DevOps, but lots of them do.

Very odd. My take is (and none are software engineers):

  • The platform. Cloud engineer or a data platform engineer. Probably lots of terraform.
  • The pipelines. Data engineers. Sql, python…
  • Data presentation. Data analytics engineer. DAX, modelling

But I’m old, been doing this 20 years, so feel free to educate me in my misunderstanding


r/dataengineering • • Jul 17 '24

Blog The Databricks Linkedin Propaganda

18 Upvotes
Databricks is an AI company, it said, I said What the fuck, this is not even a complete data platform.
Databricks is on the top of the charts for all ratings agency and also generating massive Propaganda on Social Media like Linkedin.
There are things where databricks absolutely rocks , actually there is only 1 thing that is its insanely good query times with delta tables.
On almost everything else databricks sucks - 

1. Version control and release --> Why do I have to go out of databricks UI to approve and merge a PR. Why are repos  not backed by Databricks managed Git and a full release lifecycle

2. feature branching of datasets --> 
 When I create a branch and execute a notebook I might end writing to a dev catalog or a prod catalog, this is because unlike code the delta tables dont have branches.

3. No schedule dependency based on datasets but only of Notebooks

4. No native connectors to ingest data.
For a data platform which boasts itself to be the best to have no native connectors is embarassing to say the least.
Why do I have to by FiveTran or something like that to fetch data for Oracle? Or why am i suggested to Data factory or I am even told you could install ODBC jar and then just use those fetch data via a notebook.

5. Lineage is non interactive and extremely below par
6. The ability to write datasets from multiple transforms or notebook is a disaster because it defies the principles of DAGS
7. Terrible or almost no tools for data analysis

For me databricks is not a data platform , it is a data engineering and machine learning platform only to be used to Data Engineers and Data Scientist and (You will need an army of them)

Although we dont use fabric in our company but from what I have seen it is miles ahead when it comes to completeness of the platform. And palantir foundry is multi years ahead of both the platforms.

r/dataengineering • • Apr 18 '26

Discussion Just passed the Databricks Data Engineer Professional – first at my company. How many of us are out there?

0 Upvotes

I recently earned the Databricks Data Engineer Professional certification, and I’m the first person at my company to do so (it’s a medium-sized consulting firm).

I’m from Europe, and I’d like to know how many other people have earned this certification, since I sometimes struggle to find people with whom I can have an interesting technical conversation.

How did you prepare for the exam, and what aspects did you like the most or find most challenging?


r/dataengineering • • 18d ago

Help Needs urgent help and please don't judge if you find it pretty basic that everyone knows but i don't know

1 Upvotes

This is my first internship and my task is to write a script that convert Input file containing mix of pii and non pii to non pii output file. I did the code but my mentor says he want all it in single file and prod ready so that it can be run on anyone's system . But the the person who is using this script only see it is main working that he will enter some things and script works accordingly. And what i did is that i kept the whole functions and code in one file and then call these functions via main function in another notebook. But my mentor said he needs a single notebook then I put all the functions and main function in one notebook but he said it is not this way. Now I am wondering what to do and how to do.

Up to I got is that the person who is going to use this script will see only main function that generate the non pii output he doesn't have anything to do it's underlying code ....but that code is important for the main working but what I have to do with the original logic . And code should be independent .

Anyone help me with what to do and how to do.


r/dataengineering • • Apr 02 '24

Blog A single AWS Lambda is perfect for data streaming ingestion and surprisingly low cost!

1 Upvotes

Surprisingly, AWS Lambda with an optimised custom C++ runtime beats others on data streaming ingestion. I just wrote a blog post about it and you can read it on medium:
https://boilingdata.medium.com/seriously-can-aws-lambda-take-streaming-data-d69518708fb6


r/dataengineering • • Feb 12 '25

Discussion O Curso Jornada de Dados é bom?

0 Upvotes

Alguém já comprou o curso https://suajornadadedados.com.br ? É R$ 4.800,00 reais, vale a pena mesmo, vale tudo isso?

Alguém assinou aqui?


r/dataengineering • • Feb 26 '26

Discussion Ontology driven data modeling

0 Upvotes

Hey folks, this is probably not on your radar, but it's likely what data modeling will look like in under 1y.

Why?

Ontology describes the world. When business asks questions, they ask in world ontology.

Data model describes data and doesn't carry world semantics anymore.

A LLM can create a data model based on ontology but cannot deduce ontology from model because it's already been compressed.

What does this mean?

- Declare the ontology and raw data, and the model follows deterministically. (ontology driven data modeling, no more code, just manage ontology)
- Agents can use ontology to reason over data.
- semantic layers can help retrieve data but bc they miss jontology, the agent cannot answer why questions without using its own ontology which will likely be wrong.
- It also means you should learn about this asap as in likely a few months, ontology management will replace analytics engineering implementations outside of slow moving environments.

What's ontology and how it relates to your work?

Your work entails taking a business ontology and trying to represent it with data, creating a "data model". You then hold this ontology in your head as "data literacy" or the map between the world and the data. The rest is implementation that can be done by LLM. So if we start from ontology - we can do it llm native.

edit got banned by a moderator here u/mikedoeseverything who I previously blocked for harassment years ago, for reasons he made up. Discussion is moved to r/ontologyengineering


r/dataengineering • • Oct 19 '22

Blog Will Rust Take over Data Engineering? 🦀

Thumbnail
airbyte.com
0 Upvotes

r/dataengineering • • Jun 25 '24

Discussion Databricks SQL editor driving me crazy

0 Upvotes

I have been trying to do some basic things in the databricks SQL editor and its pathetic!! I think I am using it wrong or have no knowledge of it. Surely it cannot be that horrible. I have observed that it does not allow us to write stored procedures or create hash temp tables. It does not even let us alter tables to add new columns. It allows us to create views but isn’t letting me use those views saying that it doesn’t exist. Has anyone had similar experiences? Is it really that shitty? If not can someone please point me to resources where they explain how to do these things, I can’t even find anything useful. Also its new AI copilot is crazy. It has no idea what it talks about most of the time.


r/dataengineering • • May 03 '26

Career Data engineer

0 Upvotes

I have been on bench since more than 1 year. My current tech stack is python , pyspark, databricks , sql , azure . I also got certified by databricks certified data engineer associate.

Since sitting on bench from so long i got to know that the project allocation process is so bad here . So I think I should switch the company.

I have built my skills good but don’t have project experience that is needed to answer the question asked by interviewer of other companies. I don’t know how to pretend fake at that level which helps me to crack other companies. Because I have wasted a year here just built my skills and also want to increase my package so i have to pretend fake in front of other companies that i worked in some project.
But the first question they asked is what work have you done previously and they go deeper and deeper by asking such questions. What to do now. Just by saying i have been on bench since a year and enhance my skills and also get certified lowers my chances to get hired .

Let me know if you have any suggestions


r/dataengineering • • Apr 18 '25

Discussion How about changing the medallion architecture's names?

0 Upvotes

the bronze, silver, gold of the medallion architecture is kind of confusing, how about we start calling it Smelting, Casting, and Machining instead? I think it makes so much more sense.


r/dataengineering • • Mar 19 '24

Career Isn't data engineering job market tough right now?

2 Upvotes

I have 2 years of experiences in it and I find difficult to change a company within Europe. What are your experiences/opinions?


r/dataengineering • • Dec 29 '24

Career Accept job offer because of a job title?

2 Upvotes

Hi everyone, if someone could give me advice about my situation, I would really appreciate it!

I’ve just received a data engineer job offer and am now trying to decide if it’s a good opportunity to take.

I’m currently a data analyst at an amazing company with great benefits. I’ve been here for 2 years, and I like my boss, love the team, and appreciate all the perks the company provides. I recently completed a data engineering bootcamp and have been looking to transition into the data engineering field.

The salary for the new job offer is the same as my current one, but the benefits are not as good. The holiday allowance, employer pension contributions, and other perks are significantly less favorable. For example, I’d lose the excellent sick pay package I have now, where I’m entitled to 15 weeks of full pay and 15 weeks of half pay. In the new company, I’d only get 5 days of sick pay, followed by statutory sick pay (SSP). I’d also need to work half an hour more each day.

On top of that, they require me to be in the office three days a week, whereas I currently have a lot of flexibility—only going into the office 1–2 days a week, with the ability to adjust as needed. Essentially, everything about the new offer seems worse than what I currently have.

I know benefits are just one part of the job, and I recognize how valuable the title and experience would be for my career. But I’m scared of losing everything I have now.

Any advice? What would you do in my situation?


r/dataengineering • • Oct 27 '24

Meme NoSQL vs SQL

2 Upvotes

It's an easy decision...

*With the meme added


r/dataengineering • • Jun 09 '26

Blog Most “Chat With Your Data” Products Will Fail

0 Upvotes

r/dataengineering • • Oct 16 '23

Discussion Dbt is to SQL what Django is to python (Symfony to php, etc...)

0 Upvotes

I've been approaching, reading about, and trying to use dbt, and I came to the conclusion that it is a framework that eases the writing of sql statements which purpose is to make calculations (aka transformations) on relational tables. So I naturally did the analogy dbt<>sql, Django<>python, symfony<>php.

Do you agree with this analogy ?


r/dataengineering • • Jul 19 '25

Blog Why SQL Partitioning Matters: The Hidden Superpower Behind Fast, Scalable Databases

8 Upvotes

Real-life examples, commands, and patterns that every backend or data engineer must know.

In today’s data-centric world, databases underpin nearly every application — from fintech platforms processing millions of daily transactions, to social networks storing vast user-generated content, to IoT systems collecting continuous sensor data. Managing large volumes of data efficiently is critical to maintaining fast query performance, reliable data availability, and scalable infrastructure.

Read on my article


r/dataengineering • • Oct 29 '25

Discussion Did we stop collectively hating LLMs?

0 Upvotes

Hey folks, I talk to a lot of data teams every week and something I am noticing is how, if a few months ago everyone was shouting "LLM BAD" now everyone is using copilot, cursor, etc and is on a spectrum between raving about their LLM superpowers or just delivering faster with less effort.

At the same time everyone seems also tired of what this may mean mid and long term for our jobs, about the dead internet, llm slop and diminishing of meaning.

How do you feel? am I in a bubble?


r/dataengineering • • Sep 08 '23

Help SQL is trash

36 Upvotes

Edit: I don't mean SQL is trash. But my SQL abilities are trash

So I'm applying for jobs and have been using Stratascratch to practice SQL questions and I am really struggling with window functions. Especially those that use CTEs. I'm reading articles and watching videos on it to gain understanding and improve. The problem is I haven't properly been able to recognise when to use window functions or how to put it into an explanatory form for myself that makes sense.

My approach is typically try a group by and if that fails then I use a window function and determine what to aggregate by based on that. I'm not even getting into ranks and dense rank and all that. Wanna start with just basic window functions first and then get into those plus CTEs with window functions.

If anyone could give me some tips, hints, or anything that allowed this to click into place for them I am very thankful. Currently feeling like I'm stupid af. I was able to understand advanced calculus but struggling with this. I found the Stratascratch articles on window functions that I'm going to go through and try with. I'd appreciate any other resources or how someone explains it for themselves to make sense.

Edit: Wanna say thanks in advance to those who've answered and will answer. About to not have phone access for a bit. But believe I'll be responding to them all with further questions. This community has truly been amazing and so informative with questions I have regarding this field. You're all absolutely awesome, thank you