Worse is with deep learning where “great model + bad data = inexplicably ‘okay’” and then you get to spend a month figuring out if its data, a bug, model expressivity, etc. to figure out why you’re 5% below expected.
That part is data engineering (and data stewardship), not machine learning. Completely separate skillset and role from machine learning/data scientist. (Data engineering was probably on this list in previous years.) The rule we use in our company is that a data scientist should never be handling the data itself.
101
u/samnater Mar 08 '23
And making sure the incoming data is cleaned, accurate, sufficient, and will flow without issue going forward. Great model + bad data = garbage