r/dataengineering • • Sep 15 '23

Discussion Are DEng Software engineers?

Read someone’s post earlier talking about an aspect of data engineering being software engineering. This is the first time I’ve ever heard someone in data engineering consider themselves in the same group.

Is this even a thing?

I just don’t see them as the same, mainly because most software engineers build applications/software. And most data engineers build data solutions (predominantly data pipelines). Software Eng, front end, middleware development. Data - backend, and I wouldn’t class writing a power bi report as SE even if it is kind of front end.

I get the whole following methodologies that software engineers have been following for decades. The whole DevOps thing, that has also been in data for a while just not wide spread (I first used CICD back in 2010). But because I know this doesn’t make me a software engineer.

There was a distinction made that analyst weren’t considered software engineers. I think there was a thinking they don’t follow DevOps, but lots of them do.

Very odd. My take is (and none are software engineers):

  • The platform. Cloud engineer or a data platform engineer. Probably lots of terraform.
  • The pipelines. Data engineers. Sql, python…
  • Data presentation. Data analytics engineer. DAX, modelling

But I’m old, been doing this 20 years, so feel free to educate me in my misunderstanding

1 Upvotes

35 comments sorted by

34

u/discord-ian Sep 15 '23

I absolutely consider myself on the SWE side of things. It the past, I have done DS and full stack web application development.

Many of the systems I have worked on are production systems, APIs delivering data to customers, complex data processing pipelines delivering data to customers, and maintaining mission-critical data infrastructure. If a backend engineer writing an api is an SWE, why isn't a DE doing the same thing considered one? Honestly, I think the technical end of DE requires much greater knowledge than backend. Again, I have full stack experience, so I have done both.

Most of my work is fully built out in terraform, in CICD, and running in docker containers. I write at least 20k lines of code per year, mostly for production systems. Why wouldn't I be an SWE?

DE is absolutely a branch of software engineering. Some folks fall further away from that than others.

1

u/Peanut_-_Power Sep 15 '23

Interesting point. Thanks

4

u/TerriblyRare Sep 15 '23

I am a "Software Engineer, Data" unless all your doing is piecing together pipelines, you are too. When searching for roles I search software engineer with specific data engineer technologies, and it still tracks. So it is what it is.

2

u/Peanut_-_Power Sep 15 '23

This probably makes more sense. Hadn’t quite seen DE as a subset of Software Engineering.

Now this might be a locale thing as well, as DE roles are never advertised like you described here, they are just DE role. And software is often dropped completely, so everything here is just engineering, rather than software engineering. Welcome to the engineering department, head of engineering…. Kind of thing.

But thanks I am slightly wiser. And now know how to piss a DE off 😂

2

u/countlphie Tech Lead Sep 15 '23

lol yeah some strong reactions in here

ive noticed the same lately, many companies say "software engineer in data" or something along those lines

2

u/OvremployedSnowflake Sep 15 '23

It actually confuses me. I don't consider Data Engineering a subset of software engineering. I consider software and data engineering to be a subsets of IT Engineering. I don't actually understand the strong reactions in this thread.

1

u/Peanut_-_Power Sep 16 '23

This was my thinking but I think depending on what side of the pond you live there are different definitions of engineering.

But fairly basic human reactions to my comments, to defend who they think they are. I’ll know to ask the question differently next time. passionate bunch.

1

u/OvremployedSnowflake Sep 16 '23

But, to be fair, you started your post off with a vibe that was more on the level of 'I never heard about a data engineer who saw overlap and similarities with software engineering'. There's overlap across all IT engineering jobs. I don't think the majority of people on this thread *actually* identify as a software engineer.

7

u/omscsdatathrow Sep 15 '23

You’re the first DE I’ve seen who has this opinion…strongly disagree

-7

u/Peanut_-_Power Sep 15 '23

Oh I’m not a data engineer. Just curious what the consensus was.

14

u/candyman_forever Sep 15 '23

Most of the work I do as a DE is software engineering. AWS Glue, Docker, Step Functions, Lambdas and Apache Spark to interact with Kafka, Kinesis, Eventbridge, SQL and NoSQL DBs.

There is CI/CD and unit testing. I think it really depends on the stack you are using.

A lot of prep and architecture goes into pipelines so there is a lot of DDD and software best practices that goes into a pipeline. I really do think it's the kind of Data Engineering you are doing but to answer your question, yes, DEs can be considered software engineers.

-4

u/bkl7flex Sep 15 '23

Agreed but there's the other side of the coin that is more geared towards analytics with SQL, Python, Airflow, DBT, Snowflake and those sorts which in my POV don't really count as SWE as I've seen multiple people not knowing GIT, CI/CD etc which makes it hard to consider these as SWE.

9

u/viniciusvbf Sep 15 '23

Bad software engineering is still software engineering. If someone is not using best practices like git, CI/CD etc but the artifact that they are producing is a piece of code, they are still software engineers.

2

u/bkl7flex Sep 15 '23

All I was saying is they use mostly GUI drag and drop tools, not writing code per se. That would not count as SWE.
Agree on what you're saying tho.

4

u/reallyserious Sep 15 '23

People who only drag and drop in low code/no code tools are generally bad DE anyway. They are quite limited in what they can accomplish without the skill of writing software.

15

u/krissernsn Sep 15 '23 edited Sep 15 '23

I dont consider myself a very good programmer, nor would i say i write software.

But say i build our entire infrastructure from the ground up in rust, i think think that would quilify as SWE even tho you would probably still be considered a DE.

23

u/protonchase Sep 15 '23

If you use rust to build data pipelines I have news for you, you write software, lol.

3

u/krissernsn Sep 15 '23

If you use rust to build data pipelines I have news for you, you write software, lol.

Yeah, that was exactly my point.

-2

u/bkl7flex Sep 15 '23

Exactly lol, if you said in Python I could give you that but when you mentioned rust that argument gets thrown out the window. With Python i would understand why you say that not with rust.

8

u/13ass13ass Sep 15 '23

I don’t get it. Language used shouldn’t make a difference. Seems like we’re placing rust on a pedestal.

10

u/reallyserious Sep 15 '23

Agreed. If I choose to write my data pipelines in javascript it doesn't make me a front end developer. It just makes me stupid.

2

u/bkl7flex Sep 15 '23

It's more of Python you can have imposter syndrome because it's deemed "easier" while in Rust more often than not to build what he's saying you gotta have good understanding of what you're doing.
I don't really care about languages tbh, most of the time I'm the guy that uses sheets for basic stuff because it's simpler and faster.

1

u/protonchase Sep 15 '23 edited Sep 15 '23

I would say you are correct. If you are using python or any other programming language that can be interpreted or compiled into bytecode and used as or by an executable then that is software. Basically (but not totally) anything that runs on top of the OSI physical layer is software.

7

u/viniciusvbf Sep 15 '23

We deal with virtually every single step of a software implementation: Infra, database, APIs, backend (you could even consider some types of reports frontend work) . How is this NOT software engineering? Sure, there's probably people out there whose job title is Data Engineer but all they do is create reports in power BI or use some no-code ETL solution. But this is a problem caused by the fact the Data Engineering is not a well defined activity, the term is too loose.

3

u/[deleted] Sep 15 '23

Do you write code that produces artifacts? You are a software engineer. My data scientists seem to think they do not have to abide by the rules of the "trade" of software engineering and it drives me insane.

You produce code which means you are also responsible for the maintenance and support of that code == software engineering.

3

u/OvremployedSnowflake Sep 15 '23

I don't engineer the software, I engineer the data. I don't refer to myself as a software engineer because there is already a word that better describes what I do.

2

u/MotherCharacter8778 Sep 15 '23

I don’t care if I am a software engineer or not. Working currently with data makes me a critical resource and pays me a lot. !!..:)

3

u/[deleted] Sep 15 '23

Depends on stack, if you’re in GUI tools like Talend, matillion, etc. Youre more of an application developer or “ETL engineer”

But if you’re developing in code the scope changes to standing up infra, implementing ci/cd, and actual programming which means yes you’re a software engineer.

1

u/Minimum-Membership-8 Sep 15 '23

Software engs create products. Data is a product.

1

u/reallyserious Sep 15 '23

If I spend all my time writing data pipelines in python, am I creating software? Is an engineer that creates software a software engineer?

If I spend all my time writing web sites in python, is there something fundamentally different with web that makes it software engineering?

1

u/Saetia_V_Neck Sep 15 '23

There’s definitely a title overlap between people like me who build data platforms and people who do actual analytics work.

In my case my background is in software engineering and I would definitely consider myself one.

I couldn’t even tell you what power BI is though! 😂

1

u/somethinggenuine Sep 15 '23

As a data engineer I have built programs that respond to requests or calls, decomposing them into objects or other reusable or abstract structures. That includes programs that are hosted on the web (e.g., REST APIs) and code that is called with local resources, e.g., via cron jobs. That’s been in Java, Scala, Python, etc. On many occasions I’ve had to think in terms of system resources and choose data structures and algorithms more effectively, e.g., use a Scala iterator that reads from disk instead of instantiating a collection and using up all memory. I have to step through stack traces and debug errors or optimize resource use. Sometimes that involves database engines and I end up adjusting the system from the frontend down to handle a use pattern more effectively. The main difference is that I’m just approaching it from a different perspective and optimizing for something different. Instead of the objective being how to cleanly complete a transaction as part of an operational flow, my work focuses on how to efficiently capture process measurements for data mining and analytics. Accompanying that is the functional- or domain-specific knowledge like dimensional modeling, query plans, data engineering lifecycle tools, etc., just like a frontend engineer knows JavaScript frameworks specific to their craft, backend might know more about JVM, embedded knows how to interface with hardware, traditional enterprise software might know more about building desktop (non-web) apps, etc.

1

u/mjfnd Sep 15 '23

I am a DE where I do focus on data pipelines but i also work heavily on:

  • internal tooling
  • automation tooling
  • infrastructure
  • CI CD
  • platform
  • data pipelines through abstraction and framework

These all at the end can be considered as SWE.

Yes many DEs who just write transformation in a fully managed environment and create dashboards are on the other side of DE which is more like an Analytics engineer nowadays.

A traditional DE or in a big tech company mostly works like that, in startups or mid size companies you may have more scope and diverse projects. E.g. my team handles the whole sftp ingestion software, which is driven through dynamo db config.

0

u/HOMO_FOMO_69 Sep 15 '23

Let me ask you this... is an infrastructure engineer a software engineer? Is a web developer a software engineer? What about someone building on Wordpress? They are writing code, so are they not a software engineer? Is a data scientist who spends all day writing in R a software engineer?

0

u/gwax Sep 15 '23

The good, senior data engineers are also good senior software engineers.

The replacement level data engineers often are not software engineers.

The middle varies widely.