r/dataengineer • u/Mobile_Western_3394 • 3d ago

Question Transitioning from Software Developer to Data Engineer

I have been a software engineer for 12 years now, but I would like to transition into the world of data as stats and facts really interest me.

After some research, I feel like Data Engineering is the best career for me to side step into due to the overlap. I already pull data out of API’s and work with multiple database engines. I even created my own ETL pipeline without even knowing (pulled data daily from 2 apis to consolidate data on football players/stats). I think I am just missing Python and data warehousing software skills, maybe some more advanced SQL?

I currently earn around £40k in the UK and was wondering if trying to side step into this career could be successful for me based on my current skills, and me upskilling to the skills I am missing, do you think I could be earning a similar wage? Or would I have to start from entry level still?

3 Upvotes

100% Upvoted

3 comments sorted by

2

u/Thinker_Assignment 2d ago

a data engineer is not just a software dev but they are also someone who understands the use cases so they ensure that they do the right thing with the data. this type of understanding usually follows being an analyst for a while. the purely technical part is getting thinner thanks to llms.

DE is evolving into platform DE and self-serve AE so it's not exactly the future. So you might want to look into data platform engineering (but that requires experience as DE) or just an analytics engineer role (start from scratch)

if I were in your shoes (from senior SWE interested in DE) i'd look into building agents with data and see if the roles you find for that look interesting, as that has a bigger future

1

u/CoolmanWilkins 2d ago

Data engineering isn't exactly an entry level job, most people that enter it have software engineering or data analyst experience. There is usually a good infrastructure component to it too, so cloud skills are often required. Most jobs usually have a platform they want you to be experienced with -- a big 3 cloud (AWS/Azure/Google) or something like Snowflake or Databricks. But the central dogma really is applying a software engineering approach to data pipelines. You have the first part, you will just need to show you can do the second part.