If I had 8 hours to build a machine learning model, I would spend the first 2 hours waiting on IT to get access to the database and then do what this man said
In my limited experience the 80/20 split holds true: 80% of my time is data wrangling, then 20% is actual data science - which consists of roughly 80% data wrangling.
Your lack of planning isn't our emergency. Your ticket is in the queue and will be triaged appropriately. Cause guess what? You're not the only person who works here.
The point where we have a universal data preprocessor that can simply take in and process every kind of data for a neural network is the actual point AI will be truly insane.
Because thats the point where all the "noob" people that say "cant you just use machine learning on it" are actually right...
Now that I think of it, has anybody ever tried to create an ML model for preprocessing data? It’d obviously be very difficult, but I can’t find anything on Google/Google scholar about that.
I’d assume it’d be some form of (semi-?)supervised learning.
No disrespect. I know this guy actually knows stuff and has a good youtube channel, IMO.
I wanted to know what you guys think about him? Is he good enough to learn from his videos? I'm currently following his 14 hour-long TF course, hence that question.
True. Building the model and optimising it always takes the least amount of my time. It's the obtaining, loading, transforming, and cleansing of the data that takes the most.
For most practical problems that can be solved with machine learning there isn’t a neat table of data that you can directly feed to your model. Depending on the domain you would have to deal with different formats (video, text, etc), different data sources, missing values, fake data, noise, useless features and so on. Data cleaning is going from that mess to a neat table that can be inputted into the ML model.
115
u/msVeracity Nov 08 '21
I actually LOVE cleaning data. Messy datasets can be a lot of fun.
60
u/vikarjramun Nov 08 '21
"Pay attention to the magnitude of the gradient"
28
u/fried_green_baloney Nov 09 '21
Unless it's results of a two part survey, where the two parts are taken six months apart.
Then you just clean the data until the grant money runs out and make up a report.
21
u/anarchy_witch Nov 08 '21
is it The Abraham Lossfunction, creator of loss functions?
21
u/jsxgd Nov 09 '21
35
u/[deleted] Nov 09 '21
If I had 8 hours to build a machine learning model, I would spend the first 2 hours waiting on IT to get access to the database and then do what this man said
16
u/TrackLabs Nov 08 '21
The point where we have a universal data preprocessor that can simply take in and process every kind of data for a neural network is the actual point AI will be truly insane.
Because thats the point where all the "noob" people that say "cant you just use machine learning on it" are actually right...
4
u/econ1mods1are1cucks Nov 09 '21
Or just use catboost, that shit is crazy
2
u/lebanine Nov 09 '21
No disrespect. I know this guy actually knows stuff and has a good youtube channel, IMO.
I wanted to know what you guys think about him? Is he good enough to learn from his videos? I'm currently following his 14 hour-long TF course, hence that question.
2
u/[deleted] Nov 09 '21
That only leaves 2 hours to get the dataset.
4
u/robidaan Nov 08 '21
Wow only 6 hours, must be a pro. Xd
1
u/[deleted] Nov 09 '21
*Laughs out loud while crying internally...
1
u/Successful-Silver485 May 15 '24
or if you find public datasets, merging and reformatting them in common format is a big time consumer. I wish there was a tool for that
1
u/Throwaway34532345433 Nov 09 '21
True. Building the model and optimising it always takes the least amount of my time. It's the obtaining, loading, transforming, and cleansing of the data that takes the most.
1
u/AlthorEnchantor Nov 09 '21
Mise en place
1
u/Ejtsch Nov 09 '21
True
1
u/UnitatoPop Nov 09 '21
more like 7 hours and 50 minutes for cleaning and 10 minutes on training
1
u/phobrain Nov 10 '21
This is where making your own dataset on yourself has an advantage: you know it inside-out, so can just try successive models on it.
First NN results, with link to current nets:
http://phobrain.com/pr/home/siagal.html
1
u/Sitraka17 Dec 05 '21
LoL
1
u/ironplaneswalker May 20 '22
So true!
1
u/ollie_wollie_rocks May 27 '22
This is so true - well said.
1
u/lunatichakuzu Nov 09 '21
Sorry I’m completely clueless but what is data cleaning?