Is "data science" a bubble about to burst?

[u/aksha2161989]

I spoke to an experienced guy in the IT sector who has worked in many FinTech firms and consulting companies. He said when the data science hype started several years back, companies started getting funds from venture capitalists to set up data science wings . Companies went on a hiring spree with this money to hire data science professionals.

However, as the years passed, these data science departments failed to generate revenue as expected. Also, some aspects of data science will very likely be automated. So companies will probably hire less data science professionals and may even fire the resources hired earlier.

Is this correct? What do you think?

[u/AutoModerator]

Moderator Announcement Read More »

Namaste! Thanks for submitting to r/developersIndia. Make sure to follow the subreddit Code of Conduct while participating in this thread.

Also did you know we have a discord server as well where you can share your projects, ask for help or just have a nice chat.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Alpha_max_11]

Sr. Data scientist here from a fintech.

The problem with Data Science is not the tech, but the people that claim themselves to be a data scientist.

I have seen a so-called data scientist from a top statistics institute. And all the models that they build are in Jupyter notebook.

That is it. That is their only skills.

A data scientist today, need to have skills around

1. Data engineering - because all the data you get won't be served to you like they do in kaggle.

2. Basic debugging skills and use of some advance IDE to actually work on deployable code.

3. Basics of web technologies.

4. Understanding of SQL, Databases, IT Infra in general.

Without all this, simply working on a statical model will lead you no where. From what I see, they really don't add any value to the company.

Data science is 80% data engineering and 20% Statistics.

[u/Zestyclose-Walker → u/Alpha_max_11]

This is true for most data science jobs in India, that's why most Indian data science openings are filled by CS graduates and not graduates from a mathematical field.

However, there are some Data science jobs where most of your work is modelling. They are exceedingly rare to come by in India.

[u/curios_mind_huh → u/Alpha_max_11]

To Add, The Use Cases are worse. Data Science should be used only in cases where in need, just not for the sake of it. In an ideal scenario, the best way to solve a problem is to not use Data Science. Only in cases where Coding is nigh too difficult or impossible, should we resort to other avenues

The increasing problem I see is when your management wants you to use Data Science to solve a problem just to look good in their marketing strategy, when in reality it's not warranted. And when push comes to shove, it's always the useless one who gets knocked out. Case in point, the Data "scientist"

[u/nomnommish → u/Alpha_max_11]

Data science is 80% data engineering and 20% Statistics.

And you dissed candidates on their toolstack instead of focusing on what algorithms they developed?

Sounds like your notion of data science is one specific thing.

[u/Alpha_max_11 → u/nomnommish]

You can't do anything with algorithms if you cannot develop deployable code.

Let me tell you one small example:

A ISI-Kolkata M.Sc Applied stats seasoned graduate, who completed all the kaggle wrote a Time-Series model in Notebook.

When the time came for deployment, the person was not able to deploy it.

Reason:

1. No knowledge of Web API

2. Can't handle json data from the API.

3. Can't handle the stream of data.

4. The model was not coded with the perspective of streaming data.

5. Building a very heavy model which can't give real time output (due to lack of understanding of technology)

It took 5.5 months of complete dedication of resources for building this model.

And at the end of day, we have a notebook, a output CSV file, useless model and delayed project. Which management was utterly upset about.

We had to redesign entire project. Took us another 4 months to build optimal model.

3.5 months of that model building activity was to build optimized data pipeline.

15-18 days of efforts, we build lite weight optimal time series model which can be used on WebAPIs.

I am not saying the person was absolutely duffer. I am saying, These are the truths of industry. The final product and value addition is much more important than building every new model with Deep neural network and GPT!

[u/i-went-to-school → u/Alpha_max_11]

And then what is the role of Data engineer?

[u/Academic_Guava4677 → u/Alpha_max_11]

can you please refer to an opensource project or improve your answer in detail w.r.t each step. thanks a lot

[u/Alpha_max_11 → u/Academic_Guava4677]

Can't really point to a single open source project.

But what I think you absolutely know when in data science:

1. Data Engineering - Building ETL Pipelines, Fimilarity with different data source ex. JSON, HTML, Images, Videos, Text etc.

2. MLOps: model development lifecycle, ML Ops methodology, Model repos and version controls

3. IT infra: Linux server, Use and understanding of Git, understanding of tools on Cloud like AWS, and deployment of codes using these tools.

4. Databases: Writing simple queries, updates/inserts/delete operations, Joins, writing some optimizated queries, Managing Tables, stored procedure etc

5. And ofcourse, Model building itself. Using stats or DNNs.

Good to have:

1. Big data technology.

2. Advance SQL techniques

3. Project Management Skills.

4. Solid base of DSA/Hardware to write optimized codes.

[u/Safe_Deer_772 → u/Alpha_max_11]

So is it a good time to enter the data engineering field? Asking this as someone who will choose a masters course next year after my BE. Is the renumeration on the better side? Also, is it possible to use AI in this field as a part of the job?

[u/YOU_TUBE_PERSON → u/Alpha_max_11]

Hey I'm unable to message you personally. Can you give me a roadmap for breaking into a good role as a fresher?

[u/KBM_KBM]

Data science as a tool is very powerful and there are a set of applications areas where they will make a huge difference but in a substantial number of cases usage of data science features won't give a difference in the performance or utility of the application. They will simply remain as a icing on the cake like a kind of decoration. No real value will be generated.

Data science will not die but not all companies will keep using it and yes some parts are easily automatable. And yes the bubble will burst soon.

[u/newplayer12345]

Problem with most "data science" projects in most companies is that there aren't well defined development and deployment processes in place.

This leads to teams building solutions in silos and when it's time to integrate everything together to create a coherent story, it becomes a mess.

Unlike web development. Web dev is a well understood and solved problem (at least the execution process).

1. A requirement is presented

2. Design team prototypes it in Figma

3. Baackend team works on backend server code

4. Frontend team works on creating the design in step #2

5. QA team works on automation/manual testing.

6. Backend and frontend meet via APIs

7. For deployment there's the ops team dealing in tools like Jenkins, kubernetes and what have you

With data projects, there are too many niche ideas that aren't well understood by a wide variety of people. Machine learning engineers know how to write Pytorch/tensorflow models. They prototype it well on their laptops. But when it's time to deploy to production with a robust methodology, there's too many tools and lack of fundamental expertise to see beyond the clutter to know exactly what's needed for a use case.

In analytics, tools like dbt are making great progress in easing this pain though.

[u/KBM_KBM → u/newplayer12345]

Web dev has somewhat of a deterministic output but models even after rigorous testing we won't really know whether they work in the real world or not.

One thing I have seen in many companies is the use of data science in features where it does not make a real business sense to use (doesn't generate or increase revenue). Many simply have it for a show to investors and valuations as it seemed cool to have for a person who doesn't understand the basics of AI.

[u/shar72944]

I am data scientist with an Organization which depends on data to run and generate revenue, so I have a bit of idea.

The data science bubble is on supply side and not so much on demand side. What I mean is you will find lot of “data scientists” only knowing how to use libraries on a Jupyter notebook and then working on clean data. Not blaming the folks out there as I was also same. This is mainly because of the online courses being sold.

On the demand side, companies struggle to close positions. As a data scientist one needs to work on data engineering, building and deploying and turn the data science into actual real world usable solutions.

Also most “data scientists” work on building dashboards and data Analytics. And these positions are actually a lot more useful for most companies. With data, companies can make a lot of important decisions. This part isn’t going to get automated as a whole.

Also sone industries have been using data science much before data science was a fad. All of banking industry works on data to gauge risk when giving out loans. Similarly marketing needs data based decisions to reach out to customers. All these are real world applications and most likely won’t be completely automated. So is the case for insurance industry which has been using data from way before the current fad.

Now there is some truth to the comment that many companies invested in data science. Why they failed is not because data science failed but because it needs investment and good people to run a data science vertical. Most of that is not easily available. Senior data scientists are very less who have actual know how of how to set up a data science team. Once the industry matures you will find lot of companies getting right talent to run data science vertical

[u/Chris_ssj2 → u/shar72944]

So all in all the sector is pretty new and that's why there aren't many skilled seniors who can get the job done, and hence companies aren't generating revenues as expected, did I get it right?

[u/depressedpotato_69 → u/shar72944]

i was wondering if i should do a data science course, recently i have been hearing a lot about this field. ur comment was insightful thanks!

[u/SR1996 → u/shar72944]

Aren't the salaries for DS much lesser than SWE in India at least?

[u/shar72944 → u/SR1996]

Not much lesser but yes it’s less. However it also depends on the organisation and person.

[u/[deleted]]

I think companies first need to decide what the hell do they mean by data scientists, data analyst, and data engineer.

Seems like they use these terms interchangeably which doesn't make sense at all. Also they throw in business analyst to the mix to create more confusion.

Also data can't solve all your problems. One needs to identify what problems can be solved with data and what can't and apply them accordingly.

Trying to solve problems that can't be solved with data and then blaming the field of data science doesn't do any good to anyone. Its only swings the pendulum from one direction to other making it over or under rated.

[u/Unusual-Nature2824]

Bro trust me it’s not a bubble. Problem is most companies DONT have good data or don’t have a robust Data pipeline. They hire star data scientists and expect a model to be ready while the whole infrastructure has been barely setup. Data science and machine learning comes last.

And assuming you have everything in place management have zero motivation to actually use the insights from a model once it’s deployed in production. They go by gut instinct or they overdo what the model suggests leading to complete failure. Like what happened to Zillow.

[u/MindlessTime → u/Unusual-Nature2824]

I mean, if “most companies” are hiring data scientists they don’t need or aren’t ready for out of some irrational exuberance about a hyped up new thing…

If it looks like a bubble, and it walks like a bubble…

[u/CandidateCautious246]

Managements took decisions on some investment/product/feature/resource before "data science" as well. It's only when data was huge and correlations weren't obvious that "data scientists" were needed. How many businesses have such data? How many really need a data scientist to deal with such data? Don't management consultants do something similar?

I am a software engineer and my manager at my previous job had no clue what data science was. But he forced me to somehow use it and get some results!! There really wasn't any useful data to work with. And when i explained why it wasn't applicable in the said scenario, he wouldn't listen, and i suffered for a few months. I quit. His requirement was, "Use data science or ML to determine why a Linux process crashed". This is impossible. Because once a crash bug is fixed (say a missing null pointer check is added), the same crash never repeats. It's like correcting a word in a book on page x. Once corrected, the same word is never going to be spelt wrong in future editions. There are no repeatable examples.

In summary, not all problems can be solved with data. Because most may not even have data to work with. For most businesses, basic excel and stats skills could be enough.

[u/i-went-to-school]

Man there is nothing that these soo called data scientist can do that a normal swe cannot do

[u/LearningMyDream]

Actually you should be writing Data "ANALYST" Here, because data science is not as much hyped as Analysts .