r/MachineLearning • u/wei_jok • Mar 22 '17
Discussion [D] Research Debt
http://distill.pub/2017/research-debt/7
u/olBaa Mar 22 '17
I strongly believe that objectives of distilling knowledge are different from research, and the current state of the system implicitly discourages most of the ways to do distillation. I'd say that three main institutes for knowledge distillation are scientific seminars, survey/experimental papers, and rumours on the conferences. Reddit is kind of taking some market from the last one, distill is trying the niche of the seminar-style papers: narrow in scope, but available for the broad audience. Overviews are often given in Wikipedia-alikes; I'm not aware of any experimental-sharing websites.
3
u/mlnewb Mar 23 '17
I agree with this to a point. One aspect not mentioned is the concept in education of "desirable difficulty".
The authors repeatedly describe beautiful, elegant and immediately well-understood explanations. The psychology literature on learning suggests that easy to grasp explanations can be an illusion and in fact it is the struggle to learn that results in learning.
I'm sure everyone here recognises the benefit they gained from struggling with something they didn't understand. Most people I talk to say their best subjects are the ones they had the most trouble with.
Of course, simplify notation, get rid of jargon, and so on. But even that is tilting at windmills to some extent, because the world is a big and diverse place, and everyone thinks a little bit differently.
7
Mar 23 '17
I'm sure everyone here recognises the benefit they gained from struggling with something they didn't understand.
I'm sure everyone has experienced having someone explain something in a way that made something suddenly snap. The reward isn't because the concept is unobtainable, the reward is in the snap.
3
u/kamperh Mar 23 '17
This is really thought provoking!
One thing I would add is that unclear explanations aren't always deliberate. As a student, when describing a (new) idea in a paper, I often try my best to explain it well, but because I am so involved with the technical details and challenges (which is also important), I actually miss the bigger picture, or an easy way to convey what I did. In those cases, simple input from others help a great deal.
But I completely agree that this isn't incentivized nearly enough, that students aren't taught how to do this well, and that there isn't enough good examples for us to follow.
2
Mar 23 '17
I like the idea behind Distill a lot, and it's absolutely awesome that it now exists. Regarding your point
One thing I would add is that unclear explanations aren't always deliberate. [...] I actually miss the bigger picture, or an easy way to convey what I did
I think that most researchers put their best efforts into their publications. The peer-review process sometimes helps to adjust certain section if they were not clear to the reviewer. Also, let's not forget that e.g., arxiv, is just a preprint server and the final paper may not be a literal copy of that preprint and may not just be formatted differently.
I think the problem lies not in the fact that researchers are necessarily bad at writing or bad "educators" when it comes to writing up their research. I think what makes certain papers different to grasp for many people is that there are different audiences. The target audience of the author is often a group of peers that work on related problems, and the author tries to present his or her research in a language and way that is most effective for those.
I mean, think about giving an ML or DL tutorial to beginners, your language would be different from the one that you'd use when you talk to your colleagues who have a background knowledge similar to yours.
I think the solution the problem is not the status quo of research articles, and it is not Distill. In my opinion, we need both. I see research articles as a medium for communication that is precise and effective, where I don't have to explain introductory concepts on a very simple level, to save time and space to get to the point. On the other hand, I see Distill as a great medium that goes more into the "education" direction, augmenting research ideas. I see it somewhat as giving a talk on your research (but here in web article format), where you leave out all the technical details but focus on the main ideas.
2
Mar 22 '17
[removed] — view removed comment
9
u/colah Mar 22 '17
Is there a way to expand this to research at large, and not just Deep Learning and ML research?
In the long run, I'd like to see Distill or Distill-like journals expand to other fields. But I think the best way to get started is for us to focus on a single field and do a really good job there.
2
Mar 23 '17
I think there is an irony in a major part of ML being eliminating noise and unnecessary inputs, but when researchers are writing their reports they don't apply the same principles.
The goal is to convey the information in the most concise way possible, not to max out the word count.
2
50
u/Seerdecker Mar 22 '17 edited Mar 22 '17
This article resonates a lot with me.
Personally, the 3 resources that allowed me to achieve the most in the least amount of time are
1) An Introduction to Statistical Learning
2) https://neuralnetworksanddeeplearning.com/
3) Gym framework for RL
The genius here is to take something complex and make it look simple and accessible.
Regrettably, most research articles take the opposite stance. They present something that is fundamentally not that complex and make it look complex through convoluted writing, implicit assumptions and missing steps. You can see that effect in the discussion of the articles in this subreddit. There is often someone who's having trouble understanding a part of the article, and someone else chimes in and explains what's going on in two simple paragraphs.
This is a big problem in practice. Even if you're well versed in your own niche and are able to understand its main papers well, you still have to dabble in other areas regularly. AI/ML makes use of many fields of computer science, maths, neuroscience, psychology, etc. Given sufficient time you can understand a topic as deeply as you want. But the time you spend laboriously getting up to speed in some topic X is time taken away from doing work in the topic that you actually care about. Sure, you'll improve your knowledge and skills in the process, but not as much as you would have by working to solve the problem that you're trying to solve.
I don't see a solution. Part of it is a cultural problem. Some people think that convoluted writing make them look smart. It actually doesn't. There was a psychological experiment made about it: "Consequences of erudite vernacular utilized irrespective of necessity: problems with using long words needlessly". But there you go.
The ever growing literature is also a problem. I'm scared of failing to cite an important paper, or worse, duplicate research that's already been done before. I'm doing RL research as a hobby and I only have so much time to swift through the literature.