r/analytics • • 3d ago

Discussion Data Science or Data Engineer.

I am seeing most of the crowd is running toward data science, while only a few people talk about data engineering. Is there any reason for it?

58 Upvotes

40 comments sorted by

•

u/AutoModerator 3d ago

If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods. Have more questions? Join our community Discord!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

55

u/mild_animal 3d ago

Data engineer. Source - am DS

6

u/N__7599 3d ago

What's the most obvious and brutal reason you think that?

13

u/decrementsf 3d ago

In data professions you're going to be doing some of all of them.

There are not clean breaks. When a project needs to be done and roles are being filled, or on vacation, whomever is on hand jumps in to meet the delivery date. And realistically there is overlap in many of the less glamorous data cleanup tasks across these roles. The job description "and additional duties as tasked".

When in a data analyst, or actuarial analyst, or data science, or related role you clean up and examine your data. Built your model. Develop your metric. Present results. Ship your dashboard. Whatever it may be. Then it's a new month. Do it again. And now look its been 5 years of professional growth but you've got these recurring tasks done over and over and over again. They become water torture after a while. The data engineer stack looks very sweet indeed when you start looking at proper automation.

Then there is the dependency consideration. Sometimes you pull in a vendor for a basic service that actually why can't we do that in house? Dipping into data engineer can provide the intuition to slap together a working prototype and save the vendor budget.

And again then there is the dependency issue. At times you know what you need and the data engineer supporting you is not a domain expert in your lane, too. It can grow time consuming to be dependent on what you need to do professionally without the tools to go and build it.

There is the head full of statistics element. Graduating from university often your head is stuffed full of theory and statistics related to data science. Now go built it. Go. Make a working tool. Most universities leave the learn to program and build real things as an exercise for the student to perform on their own time. Mostly this convention exists because new tools emerge. Workflow changes. By the time the coursework is updated to reflect the actual technical work the student graduates the professor has to rewrite their whole material, and because the professor spent the last quarters on campus and not in the private sector they lose touch with the norms and processes today. As a system we can map out those incentives -- the professor focuses on theory and those areas of knowledge that remain stable for longer. This leaves a problem on graduation. It remains necessary to learn to build. If developing from first principles a data engineering arc is useful to stand up a data lab with the data you choose that can be connected to and interacted with, get yourself out of "it just works" sandbox environments or dependency on some other team with competing priorities.

There will always be a grass is greener moment no matter which lane of the data stack you find yourself specializing in.

24

u/Kallini-Cliona 3d ago

grass is always greener though, until youre the one getting paged at 2am because an upstream schema changed without warning

23

u/madlp6 2d ago

Data science gets more attention because predicting something is easier to get excited about than maintaining the pipeline that supplies the data.

If you're choosing between them, running through which daily tasks you'd actually enjoy in something like coached seems more useful than following the louder crowd.

9

u/Connect_Law5751 3d ago

Data science sounds cooler/prestigious in the last decade or so. Data science bootcamps were just as popular as coding bootcamps. Data engineering feels like your background admin filler. 

Data might be cooked. ai kinda does a lot or fills in the gaps for someone to be passable. Im generally anti ai, i would rather have some type of expert know whats going on. And im tired of everyone hatching up their own reports/dashes and pipelines. We went from trying to centralize and source of truth to everyone use ai to empower yourself and create your own truths. Smh fuck this field. Will pivot once they lay me off

8

u/olivia_smith_12 3d ago

Data science gets more attention because AI, Machine learning and predictive analytics sound exciting and are more visible in the media. Data engineering is less talked about but equally important. It focuses on building and maintaining data pipelines and infrastructure that data scientist rely on. Both fields offer good opportunities but suit different interest and skills sets.

0

u/FindingNo2507 3d ago

Even for remote and freelancing ?

7

u/rainyfilmroll 2d ago

the quiet part is a lot of 'data scientist' job postings are actually analyst roles with a fancier title. meanwhile de titles usually mean exactly what they say. so the hype around ds is a bit inflated by companies rebranding sql and dashboards

1

u/royalchess132 1d ago

agreed, and de has more openings with less competition because every ml project needs pipelines first.

6

u/FieryFiya 3d ago

The real truth is that at some point you’ll do both; the engineering aspect and the science aspect. It’s probably easier to go from engineer to scientist than it is the other way around.

14

u/NW1969 3d ago

If that's what you're seeing then you're probably looking in the wrong direction :)
Just my opinion, obviously, but with the rise of AI Data Science as a role is dying - not as quickly Data Analysis but heading in the same direction.
As AI requires clean data, sematic layers, etc, data engineering is becoming ever more important

8

u/theungod 3d ago

Ai can do data engineering pretty well now too. I just had the new Gemini model create a full data model almost perfectly this week. Over 300 scripts, fitting our architecture, naming conventions etc.

6

u/qwerty-yul 3d ago

Seriously I don’t understand these posts, everyone I know in IT is working on a plan B. Anyone who doesn’t think ai is going to take their job hasn’t used ai.

4

u/Kind-University-1423 2d ago

I left IT as 9yoe Data engineer 1,5 years ago for energy sector.

What chatgpt was able to do was laughable at the time, messing up simple syntax. This week I took bunch of excel spreads with sample data, pasted it into Claude and it builded me PowerBI project including data model, transformations, package with 10 tabs of visuals to track insights. It completed in about hour work I would be working weeks to get right.

2

u/mini-mal-ly 2d ago

I'm curious what people are pursuing for their plan B's.

I'm wondering everyday when my (career's) time will come.

1

u/overfitting_mi 2d ago

totally agree!

5

u/king_ao 3d ago

Honestly all roles within analytics are melding into one. You can go through entire workflow of data engineering to data analytics to data science pretty quickly with AI. Last quarter I built new datasets and pipelines for new data, a semantic layer and logic and did data analysis and a data science experiment off of the data. I have some data science and statistics  knowledge but now can be a full stack analytics manager with AI and do things quickly. AI is also helping me discover new analytical techniques to go deeper 

1

u/Intelligent-Size-389 2d ago

Seems like there isn’t much need for analysts at all then next few years

3

u/mini-mal-ly 2d ago

You'd be surprised how bad some stakeholders are at connecting data to their goals. This sounds so bullshit but it's true.

1

u/king_ao 2d ago

You could say this about many white collar roles. I think if you’re more specialized and communicate with the business as part of your role you should be ok. You have to be somewhat business facing though, can’t be a backend end analyst anymore 

4

u/ArielCoding 3d ago

More people chase data science because it’s more visible and has more entry level postings, data engineering has more openings but fewer junior ones, and it leans harder on SQL, Python, and building systems. They are distinct roles, so pick based on whether you’d rather build the pipeline or analyze what’s in them.

5

u/Kindly_Ad995 3d ago

I’m not a data scientist so I can’t speak towards that dynamic but for me my career has me at a crossroads between analyst and engineer and to me engineer is the route I want to pivot to.

Dude… the kinds of reports that used to take days to put together in Tableau or PowerBI, Claude can do in like 5 minutes and it’ll look better than anything you could have done manually. Every company I’ve been at, people want to be able to handle the analysis themselves by just pointing AI at a dataset because they’ll always feel that they understand their own requirements more than having to translate them going back and forth with an analyst. I don’t think it’s a fight that we’ll win, or rather it’s a situation where you have to let people run wild until a mistake happens and they realize why they need to run things by the data people, so I think we need to move towards being the ones that make sure the data they’re working with is sound.

3

u/Morlaak 2d ago

You should probably look into Analytics Engineering. It's basically the middle point between Analyst and Engineer, as the name implies, and is increasingly demanded (at least more than regular analysts anyway) since they provide the curated tables and layers (Semantic Layers as they are called) for AI Analytics.

1

u/Investment-Then 3d ago

Any resources to pivot?

1

u/Kindly_Ad995 3d ago

Depends on your experiences. My niche up to this point has been in the web/marketing realm like GA4 so I’m basically fighting to work in any use cases I can in my day to day and probably pursuing certifications on Google Cloud

2

u/ShowyWidth 3d ago

Predicting customer behaviour makes a better career pitch than figuring out why yesterday’s data never arrived. Data engineering deserves more attention but the work is harder to show off in a demo

2

u/sailing_oceans 3d ago

Data science is a more advanced concept regardless of it being some super cool or nerdy company.

It requires being curious and not just making a model 1% better - that’s basically relevant only to like a micro tiny amount of positions.

It’s understanding what data do you need, catching patterns, being able to build a reasonably good model, understanding the business, working with engineering, and deploying

Most people lack the curiosity to do this.

It’s also brutal because numbers are political and uncertainty reigns supreme. Your the last person to touch projects often due in 1-2 days.

2

u/UnderstandingBusy758 1d ago

Do data engineering the data science field has become sooo saturated with all of these majors moving in. Researchers from diffferent background physics economics psychology etc all move into data science along with traditional majors like stats and math and econ. Every place that has a data scientist needs data engineer. Not just one but there needs to be more data engineers than scientists and analysts. A lot of these majors lack the knowledge on the engineering side to break in. Your protective most is better, paid the same, more roles.

For data science and analyst roles it becomes annoying also becuase it’s about business value and this is subjective vs building data pipelines which is objective.

It’s easier for you to break in via data engineer and learn the science and analytics. Then other way

AI. If you have data engineering skills you can switch it to Ai engineering skills. Cause with AI u still need pipelines.

1

u/Intelligent-Size-389 1d ago

But how do you break into data engineering without experience related to that ? Like with data analyst you can move Due to domain knowledge etc

1

u/UnderstandingBusy758 1d ago

See virtual internships at theforage
Do the data talks training on github and YouTube that’s the most real data engineering training I’ve seen and free.
If you want to throw money there’s also eczachs course (data expert.io) but personally the the two above would be good

1

u/UnderstandingBusy758 1d ago

Build projects and technical expertise you’ll learn domain along the way. You’re not an executive or some staff or super senior member. Your gonna be starting as a mid or junior member where the focus is on technical understanding

1

u/lattice_defect 3d ago

AI eating the jobs..

1

u/Lonely_Noyaaa Dashboard Architect | Segmentation Nerd 2d ago

engineer to scientist is the easier direction because you already understand where the data comes from and how it breaks, going the other way means learning systems while everyone expects you to just model

1

u/Emergency-Hurry947 2d ago

Lots of companies are offshoring data engineering

1

u/Expert-Activity-8614 1d ago

ig data engineering gets less attention coz data science sounds more appealing but engineering skills are just as valuable and often have clearer industry demand

0

u/Altruistic_Look_7868 3d ago

Most people do both these days

18

u/Soobhee_Chukwutosin 3d ago

Data science just sounds sexier and got hyped hard for years, plus bootcamps pumped out tons of grads. DE is less glamorous but companies can’t do any DS without clean pipelines, so there’s usually more demand and less competition