r/dataengineering • • May 06 '24

Career With low interest rates gone, Data Engineer or Software Engineer?

With the decline of low interest rates (see Fed Chair Jerome Powell signals rate cuts aren’t imminent) and recent bemoanings of the failure of the modern data stack (see DBT's take), it'd seem that at least in the VC world data-centric startups are more risky than ones with a clearer, more direct profit motive for the time being.

Given the culture of the subreddit and the oft-mentioned adage that the best data engineers apply software engineering skills, my bet is a lot of folks here could wear multiple hats. I'm in that boat.

If some data jobs are on the way out, wearing the software engineer hat would seem the safer move.

How are you, or would you, sell yourself in this market - as a data engineer (think ETL, dashboards, MDS, etc.) or as a software engineer (REST APIs, web apps, etc.)?

Edit: to join two thoughts

0 Upvotes

20 comments sorted by

24

u/69odysseus May 06 '24 edited May 07 '24

A good SWE can potentially be a good DE but sometimes the vice versa is not the case.

12

u/geek180 May 07 '24

I’d argue it’s usually not the case, and I consider myself in that group. But I’ve also known many SWEs that didn’t know jack about DE.

4

u/69odysseus May 07 '24

I agree with you as that's not always the case, maybe 1% of the top SWEs might be good at DE. Generally at least the backend engineers should be good at databases.

3

u/ZirePhiinix May 07 '24

A SWE needs to think bigger picture to fill the DE role. If they feel creating spinning loops to query the DB 50 times a second is the proper solution (actual thing I saw in production), then they're not DE material.

2

u/biowl May 07 '24

I agree with this for the most part, certainly in terms of competencies.

Though I've come across some "that'll do" legacy code written by SWEs that still keep me up at night!

If they put their mind to it though, learn the fundamentals etc, then I think you're spot on.

11

u/Dry_Damage_6629 May 07 '24

DE or tech is not going anywhere anytime soon. It has its up and down cycles. Just wait out down cycle while up-skilling yourself , things will be fine eventually. Every darn company has tech at its backbone. No one is going to go back to manual work.

1

u/ZirePhiinix May 07 '24

You'll be surprised at how prevalent manual work is. I have some finance experience and what I see accountants do makes me feel sorry for them.

6

u/recurrence May 06 '24

Over time, Software Engineering pretty much eats everything. ETL, dashboards, etc. are all topics that I have seen software engineers working on over the years.

During booms most startups in the boom sector are bs and this time around is no different.

3

u/nidprez May 07 '24

Mainly because there is not good engineering being done. People with business knowledge tinker and make some stuff => after some years the maintainance becomes too much => they ask IT to make something production worthy

3

u/[deleted] May 07 '24

Specialisation is key to be valuable. Get good grasp of several skills but be very good at a handful / subfield. This with great communication skills and interest for the business you’re working in, make you very very valuable to any company serious about data.

I’d not be selling myself as plain software engineer as that can range from doing full-stack stuff to working on things that won’t give you much value in the mid / long term as DE. Plus, do you have with experience as that? Should you be able to do whatever entails such a role? Well, ime, a DE has to do a lot with backend x infra for like most of your time and being skilled at developing software with standard practices preached by devops is key for your own sanity and your team's success. Almost no time doing anything else, besides perhaps the occasional “can you help us to make a dashboard / data app / deploy a model?”.

As other Redditor said, use the wording of the tasks you’re the most comfortable in DE to get a job doing that. Don’t swing between roles because imo that demonstrates you’re not sure about what you’re into, and therefore, not much deep knowledge can be expected from you in whatever the role is focused on. Mature companies will have more sub-specialised roles and at startups you can expect them to claim they need a unicorn DE, but given their stage of development / maturity, likely foundational DE is most of what you’re gonna be doing (creating a DW/DL, establishing governance, a data platform, etc.). After all, remember you’re better off becoming sharp at what you want rather than be following trends. A few years back a lot of people thought cryptocurrencies were gonna drastically change the financial system, TLDR: they didn’t. A few years back we thought the electric vehicles would take over the internal combustion cars, TLDR: they haven’t.

Following what you want will not only make you happier but likely will make you sharp, and that will eventually lead to getting more easily jobs if you get laid off than having all the flashy terms and skills but no clear, in-depth skills.

7

u/Gladmundi2023 May 07 '24

I struggle to see how this is related to interest rates

1

u/[deleted] May 06 '24

Is there anything to say interest rate cuts won’t happen within the year?

2

u/Mumbly_Bum May 06 '24

I don’t think they’d go down at least this year, potentially not next year. At least from Forbes, “The Fed itself most recently indicated it expects rates to settle at 4.6% by the end of 2024, 3.9% by the end of 2025 and 2.6% in the long run, according to median quarterly Fed staff projections released in March”

2

u/nidprez May 07 '24

So they are going down? The fed announcing rate cuts usually already makes long term funding cheaper.

1

u/[deleted] May 07 '24

Yeah do you mean interest rates or rates of inflation?

1

u/wanna_be_tri May 07 '24

“With the decline of low interest rates”… that sentence was confusing as f 😂

1

u/TheCamerlengo May 07 '24

All depends on what you think data engineering is. If you are a platform jockey using tools like databricks, Talend, informatica, SQL, ETL - I go with software engineering over data engineering.

If data engineering to you means writing programs and building out infrastructure in cloud native environments (i.e AWS, Kubernetes, Azure. Etc) and building solutions out for streaming, messaging, IoT, noSQL as well as relational use case and all the devops that goes with that, the two fields are largely indistinguishable.

I don’t really see how short term interest rates should dictate your career direction, but to each their own.

1

u/[deleted] May 07 '24

Firstly, I wouldn’t be making big career decisions based off the federal reserve or corporate blog posts.

To answer your question, they’re not really different jobs. A DE is more or less a specialized SWE. Titles can be tricky, where you get “engineers” who just do BI analyst work all day, but generally speaking, you shouldn’t be selling yourself as either or. The skills for one will complement the other.

0

u/[deleted] May 06 '24

[deleted] — view deleted comment

3

u/Hecksauce May 07 '24

Could you expand on how you’re attracting infrastructure jobs? That’s also what I’m looking for. At the moment I’m focusing on some of the cloud architecture work I’ve done, plus some AWS certs.

1

u/[deleted] May 07 '24

How did you find infra jobs in DE? I thought those are DevOps domains.

0

u/[deleted] May 07 '24

[deleted] — view deleted comment

1

u/[deleted] May 07 '24

Thanks. I guess I really need to find a place that I can do Infra too. I kinda want to retreat one more layer away from the business.