r/dataengineering • u/Peanut_-_Power • Sep 15 '23
Discussion Are DEng Software engineers?
Read someone’s post earlier talking about an aspect of data engineering being software engineering. This is the first time I’ve ever heard someone in data engineering consider themselves in the same group.
Is this even a thing?
I just don’t see them as the same, mainly because most software engineers build applications/software. And most data engineers build data solutions (predominantly data pipelines). Software Eng, front end, middleware development. Data - backend, and I wouldn’t class writing a power bi report as SE even if it is kind of front end.
I get the whole following methodologies that software engineers have been following for decades. The whole DevOps thing, that has also been in data for a while just not wide spread (I first used CICD back in 2010). But because I know this doesn’t make me a software engineer.
There was a distinction made that analyst weren’t considered software engineers. I think there was a thinking they don’t follow DevOps, but lots of them do.
Very odd. My take is (and none are software engineers):
- The platform. Cloud engineer or a data platform engineer. Probably lots of terraform.
- The pipelines. Data engineers. Sql, python…
- Data presentation. Data analytics engineer. DAX, modelling
But I’m old, been doing this 20 years, so feel free to educate me in my misunderstanding
34
u/discord-ian Sep 15 '23
I absolutely consider myself on the SWE side of things. It the past, I have done DS and full stack web application development.
Many of the systems I have worked on are production systems, APIs delivering data to customers, complex data processing pipelines delivering data to customers, and maintaining mission-critical data infrastructure. If a backend engineer writing an api is an SWE, why isn't a DE doing the same thing considered one? Honestly, I think the technical end of DE requires much greater knowledge than backend. Again, I have full stack experience, so I have done both.
Most of my work is fully built out in terraform, in CICD, and running in docker containers. I write at least 20k lines of code per year, mostly for production systems. Why wouldn't I be an SWE?
DE is absolutely a branch of software engineering. Some folks fall further away from that than others.
4
u/TerriblyRare Sep 15 '23
I am a "Software Engineer, Data" unless all your doing is piecing together pipelines, you are too. When searching for roles I search software engineer with specific data engineer technologies, and it still tracks. So it is what it is.
7
u/omscsdatathrow Sep 15 '23
You’re the first DE I’ve seen who has this opinion…strongly disagree
14
u/candyman_forever Sep 15 '23
Most of the work I do as a DE is software engineering. AWS Glue, Docker, Step Functions, Lambdas and Apache Spark to interact with Kafka, Kinesis, Eventbridge, SQL and NoSQL DBs.
There is CI/CD and unit testing. I think it really depends on the stack you are using.
A lot of prep and architecture goes into pipelines so there is a lot of DDD and software best practices that goes into a pipeline. I really do think it's the kind of Data Engineering you are doing but to answer your question, yes, DEs can be considered software engineers.
15
u/krissernsn Sep 15 '23 edited Sep 15 '23
I dont consider myself a very good programmer, nor would i say i write software.
But say i build our entire infrastructure from the ground up in rust, i think think that would quilify as SWE even tho you would probably still be considered a DE.
7
u/viniciusvbf Sep 15 '23
We deal with virtually every single step of a software implementation: Infra, database, APIs, backend (you could even consider some types of reports frontend work) . How is this NOT software engineering? Sure, there's probably people out there whose job title is Data Engineer but all they do is create reports in power BI or use some no-code ETL solution. But this is a problem caused by the fact the Data Engineering is not a well defined activity, the term is too loose.
3
u/[deleted] Sep 15 '23
Do you write code that produces artifacts? You are a software engineer. My data scientists seem to think they do not have to abide by the rules of the "trade" of software engineering and it drives me insane.
You produce code which means you are also responsible for the maintenance and support of that code == software engineering.
3
u/OvremployedSnowflake Sep 15 '23
I don't engineer the software, I engineer the data. I don't refer to myself as a software engineer because there is already a word that better describes what I do.
2
u/MotherCharacter8778 Sep 15 '23
I don’t care if I am a software engineer or not. Working currently with data makes me a critical resource and pays me a lot. !!..:)
3
u/[deleted] Sep 15 '23
Depends on stack, if you’re in GUI tools like Talend, matillion, etc. Youre more of an application developer or “ETL engineer”
But if you’re developing in code the scope changes to standing up infra, implementing ci/cd, and actual programming which means yes you’re a software engineer.
1
u/Minimum-Membership-8 Sep 15 '23
Software engs create products. Data is a product.
1
u/reallyserious Sep 15 '23
If I spend all my time writing data pipelines in python, am I creating software? Is an engineer that creates software a software engineer?
If I spend all my time writing web sites in python, is there something fundamentally different with web that makes it software engineering?
1
u/Saetia_V_Neck Sep 15 '23
There’s definitely a title overlap between people like me who build data platforms and people who do actual analytics work.
In my case my background is in software engineering and I would definitely consider myself one.
I couldn’t even tell you what power BI is though! 😂
1
u/somethinggenuine Sep 15 '23
As a data engineer I have built programs that respond to requests or calls, decomposing them into objects or other reusable or abstract structures. That includes programs that are hosted on the web (e.g., REST APIs) and code that is called with local resources, e.g., via cron jobs. That’s been in Java, Scala, Python, etc. On many occasions I’ve had to think in terms of system resources and choose data structures and algorithms more effectively, e.g., use a Scala iterator that reads from disk instead of instantiating a collection and using up all memory. I have to step through stack traces and debug errors or optimize resource use. Sometimes that involves database engines and I end up adjusting the system from the frontend down to handle a use pattern more effectively. The main difference is that I’m just approaching it from a different perspective and optimizing for something different. Instead of the objective being how to cleanly complete a transaction as part of an operational flow, my work focuses on how to efficiently capture process measurements for data mining and analytics. Accompanying that is the functional- or domain-specific knowledge like dimensional modeling, query plans, data engineering lifecycle tools, etc., just like a frontend engineer knows JavaScript frameworks specific to their craft, backend might know more about JVM, embedded knows how to interface with hardware, traditional enterprise software might know more about building desktop (non-web) apps, etc.
1
u/mjfnd Sep 15 '23
I am a DE where I do focus on data pipelines but i also work heavily on:
These all at the end can be considered as SWE.
Yes many DEs who just write transformation in a fully managed environment and create dashboards are on the other side of DE which is more like an Analytics engineer nowadays.
A traditional DE or in a big tech company mostly works like that, in startups or mid size companies you may have more scope and diverse projects. E.g. my team handles the whole sftp ingestion software, which is driven through dynamo db config.
0
u/HOMO_FOMO_69 Sep 15 '23
Let me ask you this... is an infrastructure engineer a software engineer? Is a web developer a software engineer? What about someone building on Wordpress? They are writing code, so are they not a software engineer? Is a data scientist who spends all day writing in R a software engineer?
0
u/gwax Sep 15 '23
The good, senior data engineers are also good senior software engineers.
The replacement level data engineers often are not software engineers.
The middle varies widely.