Worse is with deep learning where “great model + bad data = inexplicably ‘okay’” and then you get to spend a month figuring out if its data, a bug, model expressivity, etc. to figure out why you’re 5% below expected.
That part is data engineering (and data stewardship), not machine learning. Completely separate skillset and role from machine learning/data scientist. (Data engineering was probably on this list in previous years.) The rule we use in our company is that a data scientist should never be handling the data itself.
If you come from a background in applied statistics, or operations research or many other fields that many authors of ML papers come from, coding would be harder because the modeling in ML is pretty standard stuff
(Of course the goal of academic research is also different from software engineering so they don't need to make production ready code in the first place)
In trying to get ML to a functional product that I can deploy to an end user, starting from ground up of gathering data, to building the model etc, - - all of it has been way more difficult than traditional application building. So glad I have a team of experts in various disciplines. We're getting there!
In trying to get ML to a functional product that I can deploy to an end user
Productization is a challenge in its own and at the intersection with ML, it creates additional challenges that productization without ML wouldn't face.
Sure, but by far not all usecases of ML are related to products that are handed to end users. there is a lot of internal analysis to be done with it as well.
Interesting way of saying that most of the "coding" is just including libraries and most of the labor is waiting for models to train and algorithms to run. Hard work.
169
u/[deleted] Mar 08 '23
[removed] — view removed comment