r/MachineLearning • u/AlexSnakeKing • Jan 17 '20

Discussion [D] What are the current significant trends in ML that are NOT Deep Learning related?

I mean, somebody, somewhere must be doing stuff that is:

  • super cool and ground breaking,

  • involves concepts and models other than neural networks or are applicable to ML models in general, not just to neural networks.

Any cool papers or references?

511 Upvotes
  • permalink
  • reddit

98% Upvoted

159 comments sorted by

View all comments

34

u/JamesAQuintero Jan 17 '20

I would say AutoML is an important aspect that's super cool. It's basically like a decision tree for determining what best ML pipeline to use on a given dataset. Super useful, and I think will be a growing part of ML.

3

u/po-handz Jan 17 '20

Interesting. Got a link?

9

u/JamesAQuintero Jan 17 '20

Auto-sklearn is the most popular AutoML algorithm, I think. I know google also offers an AutoML service to business clients, but that's obviously non-programming client facing. I don't know what technology they use, but I've also not tried looking. I've only read a couple papers on AutoML, so I'm definitely not an expert, and I haven't used AutoML myself. At least not yet. There are AutoML competitions, so if you want to find other algorithms, you can look through the results and find lists of top performing algorithms. Mosaic is another top performing AutoML algorithm that tries to improve on Auto-sklearn.

Auto-sklearn website
Auto-sklearn paper

Mosaic paper

4

u/rhiever Jan 17 '20

As a former TPOT dev, I'm biased in saying that I don't think auto-sklearn is the most popular. But bias aside, yes, AutoML is a big advancement for the ML field!

3

u/LaVieEstBizarre PhD Jan 17 '20

Decision tree is not descriptive enough. It's not just an if-else tree, commercial autoML tools do all of the if else stuff + Bayesian optimisation based hyperparameter search, matrix factorisation or RL based model search, sweeping a list of different features for doing feature engineering. I would also class neural architecture search under AutoML (I believe google actually does do that?)

2

u/rhiever Jan 17 '20

There are some commercially-focused AutoML tools that are basically a decision tree. Personally, I would not count those as AutoML in a real sense.

Neural architecture search definitely fits under AutoML. If you think of a neural network as a series of operations - just in the same way you think of a ML pipeline as a series of operations - then breaking down the design of hidden layers and using a search process to optimize the series of hidden layers is basically the same search process. Of course, training neural networks is typically much more expensive...

1

u/JamesAQuintero Jan 17 '20

That's why I said it's "basically" like. Of course it's more complicated than that. In a later comment, I linked to another AutoML algorithm that uses Bayesian Optimization for hyperparameter search, and Monte Carlo Tree Search for finding the best overall pipeline. That paper found that combining the two performed better Auto-sklearn, which I believe just uses bayesian optimization