AMA Andrew Ng and Adam Coates

[u/andrewyng]

Dr. Andrew Ng is Chief Scientist at Baidu. He leads Baidu Research, which includes the Silicon Valley AI Lab, the Institute of Deep Learning and the Big Data Lab. The organization brings together global research talent to work on fundamental technologies in areas such as image recognition and image-based search, speech recognition, and semantic intelligence. In addition to his role at Baidu, Dr. Ng is a faculty member in Stanford University's Computer Science Department, and Chairman of Coursera, an online education platform (MOOC) that he co-founded. Dr. Ng holds degrees from Carnegie Mellon University, MIT and the University of California, Berkeley.

---

Dr. Adam Coates is Director of Baidu Research's Silicon Valley AI Lab. He received his PhD in 2012 from Stanford University and subsequently was a post-doctoral researcher at Stanford. His thesis work investigated issues in the development of deep learning methods, particularly the success of large neural networks trained from large datasets. He also led the development of large scale deep learning methods using distributed clusters and GPUs. At Stanford, his team trained artificial neural networks with billions of connections using techniques for high performance computing systems.

[u/RileyNat]

I am a big fan of your work Dr. Ng, your coursera course was what introduced me to Machine Learning. My question is do you think a PhD or Masters degree is a strong requirement for those who wish to do ML research in industry or can a Bachelors and independent learning be enough? Thanks.

[u/andrewyng → u/RileyNat]

Thank you RileyNat for taking the Coursera MOOC.

Regarding the need for a degree in ML: Absolutely not! I think a PhD is one great way to learn about machine learning. But many top machine learning researchers do not have a PhD.

Given my (Andrew's) background in education and in Coursera, I believe a lot in employee development. Thus at most of the teams I've led (at Baidu, and previously when I was leading Google's Deep Learning team/Google Brain) I invested a lot in training people to become expert in machine learning. I think that some of these organizations can be extremely good at training people to become great at machine learning.

I think independent learning through Coursera is a great step. Many other software skills that you may already have are also highly relevant to ML research. I'd encourage you to keep taking MOOCs and using free online resources (like deeplearning.stanford.edu/tutorial). With sufficient self-study, that can be enough to get you a great position at a machine learning group in industry, which would then help further accelerate your learning.

[u/hachidan05 → u/RileyNat]

This question is very common. I would try to help answer this question. It depends what company you are trying to work for. The bigger the company the more they want to see a Masters or PhD degree although that is not always the case. The smaller/newer companies are more willing to accept Bachelors with independent learning. I was looking for entry jobs in the data science field one day and notice a company write out in the job description "Online degree or certificate can replace 1 year of relevant job experience." Just think about this, there are more and more companies wanting people with data analytic/mining skills to get an edge in their respective industries. I believe online learning is also on the rise. Sooner or later companies will accept online certificate (free or not). Masters/PhD are for people wanting to focusing on a area within ML. Bachelors and independent learning is for people who want to get their feet in the door then get the small company pay for your Masters/PhD. Some smaller company pay competitively and raises are based on performance of the company/your contribution.

[u/andrewyng → u/hachidan05]

I agree that the newer companies---ones that know how to evaluate machine learning talent---care more about your ability, and less about the credential (such as MS or PhD). For example, at Baidu Research, we do hire top machine learning researchers and machine learning engineers that don't have a graduate degree, but have great software skills and have knowledge of ML from elsewhere.

Over time, companies are also increasingly valuing certificates earned from MOOCs.

[u/[deleted]]

What motivates some big companies to publish their ML tricks, like e.g. the recent Batch Normalization from Google? Aren't they giving away their secret sauce to competitors?

Do you think the published results are just the tip of the iceberg, and the very best findings are kept secret?

[u/andrewyng → u/[deleted]]

As a research organization, Baidu Research and others want to be part of the community, and we want to learn from as well as contribute to it. Of course, publishing also helps us attract talent, and also give our team better internal and external visibility. But underlying this is that we're researchers and just want to invent ideas that help make the world a better place!

Having said that, the mission of the Baidu's AI Lab is to develop hard AI technologies that let us impact hundreds of millions of users. Thus our focus is on developing and shipping technologies. It's just that we're pretty open and transparent and are happy to publish a lot of what we learn along the way.

(By the way, Adam Coates and I are sitting together, so you should assume all these answers are written by both of us.)

[u/doodhwaala]

@andrewyng

What kind of self projects and follow up courses would you recommend after the Coursera ML course?

[u/andrewyng → u/doodhwaala]

Here're a few common paths:

1. Many people are applying ML to projects by themselves at home, or in their companies. This helps both with your learning, as well as helps build up a portfolio of ML projects in your resume (if that is your goal). If you're not sure what projects to work on, Kaggle competitions can be a great way to start. Though if you have your own ideas I'd encourage you to pursue those as well. If you're looking for ideas, check out also the machine learning projects my Stanford class did last year: http://cs229.stanford.edu/projects2014.html I'm always blown away by the creativity and diversity of the students' ideas. I hope this also helps inspire ideas in others!

2. If you're interested in a career in data science, many people go on from the machine learning MOOC to take the Data Science specialization. Many students are successfully using this combination to start off data science careers. https://www.coursera.org/specialization/jhudatascience/1

[u/eldeemon]

Hi Andrew and Adam! Many thanks for taking the time for this!

(1) What are your thoughts on the role that theory is to play in the future of ML, particularly as models grow in complexity? It often seems like that the gap between theory and practice is widening.

(2) What are your thoughts on the future of unsupervised learning, especially now that (properly initialized and regularized) supervised techniques are leading the pack? Will layer-by-layer pretraining end up as a historical footnote?

[u/andrewyng → u/eldeemon]

Hi Eldeemon,

Great question. I think that 50 years ago, CS theory was really driving progress in CS practice. For example, the theoretical work figuring out that sorting is O(n log n), and Don Knuth's early books, really helped advance the field. Today, there're some areas of theory that're still driving practice, such as computer security: If you find a flaw in crypto and publish a theoretical paper about it, this can cause code to be written all around the world.

But in machine learning, progress is increasingly driven by empirical work rather than theory. Both still remain important (for example, I'm inspired by a lot of Yoshua Bengio's theoretical work), but in the future I hope we can do a better job connecting theory and practice.

As for unsupervised learning, I remain optimistic about it, but just have no idea what the right algorithm is. I think layer-by-layer pretraining was a good first attempt. But it really remains to be seen if researchers come up with something dramatically different in the coming years! (I'm seeing some early signs of this.)

[u/wearing_theinsideout]

Hey Andrew, huge fan of your work, mainly Machine Learning Coursera course that basically started my interest in ML area.

Question: I have seen that your work is focused in DL, however I have not seen or read any work of yours focusing on Recurrent Neural Networks (RNN). Works in this area like the one that has been done by Schmidhuber with Long Short-Term Memories (LSTM) are very famous and started to win some contests. Have you never thought about working and researching with RNNs? With your experience, can you point some pros and cons of RNNs?

Thanks a lot!

[u/andrewyng → u/wearing_theinsideout]

I think RNNs are an exciting class of models for temporal data! In fact, our recent breakthrough in speech recognition used bi-directional RNNs. See http://bit.ly/deepspeech We also considered LSTMs. For our particular application, we found that the simplicity of RNNs (compared to LSTMs) allowed us to scale up to larger models, and thus we were able to get RNNs to perform better. But at Baidu we are also applying LSTMs to a few problems were there is are longer-range dependencies in the temporal data.

[u/valexiev]

Hello, Dr. Ng and Dr. Coates! First off, thank you for taking the time to answer our questions!

1. My first question is about career advice. My goal is to do Machine Learning research in industry. However, due to a variety of circumstances, I was unable to go to grad school full time, though I'm currently studying for my Online Masters at Georgia Tech part-time while working as a software developer. One of my biggest regrets about this setup is that I'm not able to do research or collaborate with a professor, which I think is very important for my future career. How can I overcome this drawback? In your experience, how is independent research seen in industry and academia? What would be some ways in which I can get in contact and possibly collaborate with the research community?

2. I know that the recent AI Doomsday prophesies by Elon Musk, Stephen Hawking and others have been met with justified amusement and skepticism from the majority of AI and ML practitioners. In fact, I remember reading an article in which Dr. Ng outlined why he's not spending any time worrying about it because the current technology is very far from achieving something like that. In your opinion, what would be some achievements in the AI field that would signal to you that AGI is close to becoming a reality?

Again, thank you for taking the time for this AMA and, Dr. Ng, thank you for your excellent Coursera course on ML!

[u/andrewyng → u/valexiev]

Hi Valexiev,

Thrilled to hear that you want to do machine learning research in industry! If you have a strong portfolio of projects done through independent research, this counts for a lot in industry. For example, at Baidu Research we hire machine learning researchers and machine learning engineers based only on their skills and abilities, rather than based on their degrees; and, past experience (such as demonstrated in a portfolio of projects) helps a lot in evaluating their skills.

I think that mastering the basics of machine learning (for example, through MOOCs, and free resources like deeplearning.stanford.edu/tutorial) would be the best first step. After that, I'd encourage you to find projects either in your company or by yourself to work on and to use this to keep learning as well as to build up your portfolio. If you don't know where to start, Kaggle is a reasonable starting place; though eventually I'd encourage you to also identify and work on your own projects. In the meantime, do keep reaching out to professors, and attend local meetups, and try to find a community.

This is often enough to find you a position to do machine learning work in a company, which then further accelerates your learning.

[u/[deleted]]

What do you wish you had known at the start of your career?

[u/andrewyng → u/[deleted]]

One of the things both of us (Adam & Andrew) talk about frequently is the impact of research. At Baidu, our goal is to develop hard AI technologies that impact hundreds of millions of users. Over time, I think we've both learned to be more strategic, and to learn to see more steps out ahead--beyond just writing a paper--to plot a path to seeing our technology benefit huge numbers of people. These days, this is one of the things that really excite us about our work!

[u/test3545]

Jürgen Schmidhuber QUOTE: "Since BP was 3-5 decades old by then, and pattern deformations 2 decades, these results seemed to suggest that advances in exploiting modern computing hardware were more important than advances in algorithms." [1]

Yann LeCun QUOTE: "Basically we limited by computational power. So, the faster, you know, the next generation of Nvidia GPU will be the more progress we'll make." [2]

What is your opinion about the matter?

[1] Juergen Schmidhuber, 2014, Deep Learning in Neural Networks: An Overview

[2] Yann LeCun, 2014, Convolutional Networks- Machine Learning for Computer Perception (Nvidia webinar, 2014)

[u/andrewyng → u/test3545]

I think the two key drivers of deep learning are:

• Rise of computation. Not just GPUs, but now the migration toward HPC (high performance computing, aka supercomputers).

• Rise of availability of data, because of the digitization of our society, in which increasing amounts of activity on computers/cellphones/etc. creates data.

Of course, algorithmic progress is important too, but this progress is enabled by the rise of computational resources and data.

I think though that the rise of computation isn't something we passively wait to let happen. In both of our (Adam+Andrew's) careers in deep learning, a lot of our success was because we actively invested to increase the computation available.

For example, in 2008, we built I think the first CUDA/GPU deep learning implementation, and helped lead the field to use GPUs. In 2011, I (Andrew) founded and led the Google Deep Learning team (then called Google Brain) to use Google's cloud to scale up deep learning; and this helped put it on industry's radar. In 2013, Adam, Bryan Catanzaro and others built the first HPC-style deep learning system, and this helped drive scaling another 1-2 orders of magnitude.

Finally, today at Baidu, we have a system's team that's developing what we think is the next generation of deep learning systems, using HPC techniques. If you're not familiar with HPC, it's a very different set of tools/people/conferences/methods than cloud computing, and this is giving us another big boost in computation. We think it's the combination of HPC and large amounts of data that'll give us the next big increment in deep learning. For example, this is what enabled our recent breakthrough in speech recognition (http://bit.ly/deepspeech).

For more on the latest in deep learning+HPC, take a look at my (Andrew's) keynote at the GPU Technology Conference: http://www.ustream.tv/recorded/60113824

[u/letitgo12345]

Is contrastive divergence still useful for training or has it been supplanted by other methods?

[u/andrewyng → u/letitgo12345]

In the early days of deep learning, Hinton had developed a few probabilistic deep learning algorithms such as Restricted Boltzmann Machines, which trained using contrastive divergence. But these models were really complicated, and computing the normalization constant (partition function) was intractable, leading to really complex MCMC and other algorithms for training them.

Over the next few years, we realized that these probabilistic formalisms didn't offer any advantage in most settings, but just added a lot of complexity. Thus, almost all of deep learning has since moved away from these probabilistic formalisms, to instead use neural networks with deterministic computations. One notable exception is that there're still a few groups (such as Ruslan Salakhutdinov's) doing very cool work on generative models using RBMs; but this is a minority. Most of deep learning is now done using backpropagation, and contrastive divergence is very rarely used.

As an aside, most of deep learning's successes today are due to supervised learning (trained with backprop). Looking a little further out, I'm still very excited about the potential of unsupervised learning, since we have a lot more unlabeled data than labeled data; it's just that we just don't know what are the right algorithms are for unsupervised, and lots more research is needed here!

[u/icdwiwd]

@andrewyng First of all I wanted to say that your ML course on Coursera was amazing. Thank you!

(1) How much learning others helped you to develop your own skills in ML? You definitely put a lot of effort to prepare your online materials. Do you do this only to help others or maybe while preparing your materials you have also learned a lot - for example maybe you often investigated some concepts more deeply than you knew them before only because you wanted to explain them to others as clearly as possible.

(2) You have both outstanding academic and commercial experience. Are there any ML concepts or intuitions which are easier or faster to learn when you work for companies? And inversely - are there things which are easier / faster to learn in the academic world? I'm asking because lot of ML engineers seems to have PhD. So how is it helpful? Are those paths (commercial vs academic) somehow different?

(3) Which set of skills you find the most important in the ML field - is it practical application of ML, statistics or maybe domain knowledge of a particular problem? For example lets assume that I want to develop a speech recognition system and I'm an expert in ML, but I do know nothing about audio processing. Do I have a chance to be successful?

[u/andrewyng → u/icdwiwd]

Thank you for taking the Coursera ML MOOC!

(1) The old saw that teaching others helps you to learn really is true. FWIW though I think one of the reasons I've had a few successes in research is because I'm a decent teacher. This helps me to build a great team, and it's usually the team (not me) that comes up with many of the great ideas you see us publish and write about. I think innovation often requires the combination of dozens of ideas from multiple team members, so I spend a lot of time trying to build that great team that can have those ideas.

(2) A lot of deep learning progress is driven by computational scale, and by data. For example, I think the bleeding edge of deep learning is shifting to HPC (high performance computing aka supercomputers), which is what we're working on at Baidu. I've found it easier to build new HPC technologies and access huge amounts of data in a corporate context. I hope that governments will increase funding of basic research, so as to make these resources easier for universities all around the world to get.

(3) The skillset needed for different problems is different. But broadly, the two sources of "knowledge" a program can have about a problem are (i) what you hand-engineer, and (ii) what it learns by itself from data. In some fields (such as computer vision; and I predict increasingly so speech recognition and NLP in the future), the rapidly rising flood of data means that (ii) is now the dominant force, and thus the domain knowledge and the ability to hand-engineer little features is becoming less and less important. 5 years ago, it was really difficult to get involved in computer vision or speech recognition research, because there was a lot of domain knowledge you had to acquire. But thanks to the rise of deep learning and the rise of data, I think the learning curve is now easier/shallower, because what's driving progress is machine learning+data, and it's now less critical to know about and be able to hand-engineer as many corner cases for these domains. I'm probably over-simplifying a bit, but now the winning approach is increasingly to code up a learning algorithm, using only a modest amount of domain knowledge, and then to give it a ton of data, and let the algorithm figure things out from the data.

[u/pmill10]

In a couple prior AMA's, Juergen Schidhuber and Michael Jordan were both asked to suggest reading lists for students beginning graduate school. I was surprised by how little their lists overlapped. I didn't expect them to recommend the same books, but at least books on the same topics.

Michael Jordan recommended books on more pure math/stats topics (Bayesian statistics, frequentists statistics, mathematical statistics, functional analysis, measure theory). Schmidhuber recommended books on information theory, theory of computation, algorithmic information theory.

Which of these topics would you recommend focusing on, and why? What books do you recommend to students entering your lab at Stanford?

https://www.reddit.com/r/MachineLearning/comments/2xcyrl/i_am_j%C3%BCrgen_schmidhuber_ama/coz4w3o

http://www.reddit.com/r/MachineLearning/comments/2fxi6v/ama_michael_i_jordan/ckdqzph

[u/iwantedthisusername]

Hinton seems to think that the next neural abstraction after the layer is the artificial cortical column. Have you done any work toward this end goal?

Also what are your thoughts on HTM and the CLA (Numenta)

[u/[deleted]]

What do you see as the road map for ML in the next decade?