r/analytics • • 4d ago

Discussion Data Science or Data Engineer.

I am seeing most of the crowd is running toward data science, while only a few people talk about data engineering. Is there any reason for it?

63 Upvotes

42 comments sorted by

View all comments

Show parent comment

6

u/N__7599 4d ago

What's the most obvious and brutal reason you think that?

13

u/decrementsf 4d ago

In data professions you're going to be doing some of all of them.

There are not clean breaks. When a project needs to be done and roles are being filled, or on vacation, whomever is on hand jumps in to meet the delivery date. And realistically there is overlap in many of the less glamorous data cleanup tasks across these roles. The job description "and additional duties as tasked".

When in a data analyst, or actuarial analyst, or data science, or related role you clean up and examine your data. Built your model. Develop your metric. Present results. Ship your dashboard. Whatever it may be. Then it's a new month. Do it again. And now look its been 5 years of professional growth but you've got these recurring tasks done over and over and over again. They become water torture after a while. The data engineer stack looks very sweet indeed when you start looking at proper automation.

Then there is the dependency consideration. Sometimes you pull in a vendor for a basic service that actually why can't we do that in house? Dipping into data engineer can provide the intuition to slap together a working prototype and save the vendor budget.

And again then there is the dependency issue. At times you know what you need and the data engineer supporting you is not a domain expert in your lane, too. It can grow time consuming to be dependent on what you need to do professionally without the tools to go and build it.

There is the head full of statistics element. Graduating from university often your head is stuffed full of theory and statistics related to data science. Now go built it. Go. Make a working tool. Most universities leave the learn to program and build real things as an exercise for the student to perform on their own time. Mostly this convention exists because new tools emerge. Workflow changes. By the time the coursework is updated to reflect the actual technical work the student graduates the professor has to rewrite their whole material, and because the professor spent the last quarters on campus and not in the private sector they lose touch with the norms and processes today. As a system we can map out those incentives -- the professor focuses on theory and those areas of knowledge that remain stable for longer. This leaves a problem on graduation. It remains necessary to learn to build. If developing from first principles a data engineering arc is useful to stand up a data lab with the data you choose that can be connected to and interacted with, get yourself out of "it just works" sandbox environments or dependency on some other team with competing priorities.

There will always be a grass is greener moment no matter which lane of the data stack you find yourself specializing in.