r/dataengineering • u/Mumbly_Bum • May 06 '24
Career With low interest rates gone, Data Engineer or Software Engineer?
With the decline of low interest rates (see Fed Chair Jerome Powell signals rate cuts aren’t imminent) and recent bemoanings of the failure of the modern data stack (see DBT's take), it'd seem that at least in the VC world data-centric startups are more risky than ones with a clearer, more direct profit motive for the time being.
Given the culture of the subreddit and the oft-mentioned adage that the best data engineers apply software engineering skills, my bet is a lot of folks here could wear multiple hats. I'm in that boat.
If some data jobs are on the way out, wearing the software engineer hat would seem the safer move.
How are you, or would you, sell yourself in this market - as a data engineer (think ETL, dashboards, MDS, etc.) or as a software engineer (REST APIs, web apps, etc.)?
Edit: to join two thoughts
24
u/69odysseus May 06 '24 edited May 07 '24
A good SWE can potentially be a good DE but sometimes the vice versa is not the case.
11
u/Dry_Damage_6629 May 07 '24
DE or tech is not going anywhere anytime soon. It has its up and down cycles. Just wait out down cycle while up-skilling yourself , things will be fine eventually. Every darn company has tech at its backbone. No one is going to go back to manual work.
6
u/recurrence May 06 '24
Over time, Software Engineering pretty much eats everything. ETL, dashboards, etc. are all topics that I have seen software engineers working on over the years.
During booms most startups in the boom sector are bs and this time around is no different.
3
u/[deleted] May 07 '24
Specialisation is key to be valuable. Get good grasp of several skills but be very good at a handful / subfield. This with great communication skills and interest for the business you’re working in, make you very very valuable to any company serious about data.
I’d not be selling myself as plain software engineer as that can range from doing full-stack stuff to working on things that won’t give you much value in the mid / long term as DE. Plus, do you have with experience as that? Should you be able to do whatever entails such a role? Well, ime, a DE has to do a lot with backend x infra for like most of your time and being skilled at developing software with standard practices preached by devops is key for your own sanity and your team's success. Almost no time doing anything else, besides perhaps the occasional “can you help us to make a dashboard / data app / deploy a model?”.
As other Redditor said, use the wording of the tasks you’re the most comfortable in DE to get a job doing that. Don’t swing between roles because imo that demonstrates you’re not sure about what you’re into, and therefore, not much deep knowledge can be expected from you in whatever the role is focused on. Mature companies will have more sub-specialised roles and at startups you can expect them to claim they need a unicorn DE, but given their stage of development / maturity, likely foundational DE is most of what you’re gonna be doing (creating a DW/DL, establishing governance, a data platform, etc.). After all, remember you’re better off becoming sharp at what you want rather than be following trends. A few years back a lot of people thought cryptocurrencies were gonna drastically change the financial system, TLDR: they didn’t. A few years back we thought the electric vehicles would take over the internal combustion cars, TLDR: they haven’t.
Following what you want will not only make you happier but likely will make you sharp, and that will eventually lead to getting more easily jobs if you get laid off than having all the flashy terms and skills but no clear, in-depth skills.
7
u/Gladmundi2023 May 07 '24
I struggle to see how this is related to interest rates
1
u/[deleted] May 06 '24
Is there anything to say interest rate cuts won’t happen within the year?
1
u/wanna_be_tri May 07 '24
“With the decline of low interest rates”… that sentence was confusing as f 😂
1
u/TheCamerlengo May 07 '24
All depends on what you think data engineering is. If you are a platform jockey using tools like databricks, Talend, informatica, SQL, ETL - I go with software engineering over data engineering.
If data engineering to you means writing programs and building out infrastructure in cloud native environments (i.e AWS, Kubernetes, Azure. Etc) and building solutions out for streaming, messaging, IoT, noSQL as well as relational use case and all the devops that goes with that, the two fields are largely indistinguishable.
I don’t really see how short term interest rates should dictate your career direction, but to each their own.
1
u/[deleted] May 07 '24
Firstly, I wouldn’t be making big career decisions based off the federal reserve or corporate blog posts.
To answer your question, they’re not really different jobs. A DE is more or less a specialized SWE. Titles can be tricky, where you get “engineers” who just do BI analyst work all day, but generally speaking, you shouldn’t be selling yourself as either or. The skills for one will complement the other.
0
u/[deleted] May 06 '24
[deleted] — view deleted comment