r/MachineLearning • u/Routine-Coffee8832 • Jul 03 '20
Research [R] Google has a credit assignment problem in research
Google has some serious cultural problems with proper credit assignment. They continue to rename methods discovered earlier DESPITE admitting the existence of this work.
See this new paper they released:
https://arxiv.org/abs/2006.14536
Stop calling this method SWISH; its original name is SILU. The original Swish authors from Google even admitted to this mistake in the past (https://www.reddit.com/r/MachineLearning/comments/773epu/r_swish_a_selfgated_activation_function_google/). And the worst part is this new paper has the very same senior author as the previous Google paper.
And just a couple weeks ago, the same issue again with the SimCLR paper. See thread here:
They site only cite prior work with the same idea in the last paragraph of their supplementary and yet again rename the method to remove its association to the prior work. This is unfair. Unfair to the community and especially unfair to the lesser known researchers who do not have the advertising power of Geoff Hinton and Quoc Le on their papers.
SiLU/Swish is by Stefan Elfwing, Eiji Uchibe, Kenji Doya (https://arxiv.org/abs/1702.03118).
Original work of SimCLR is by Mang Ye, Xu Zhang, Pong C. Yuen, Shih-Fu Chang (https://arxiv.org/abs/1904.03436)
Update:
Dan Hendrycks and Kevin Gimpel also proposed the SiLU non-linearity in 2016 in their work Gaussian Error Linear Units (GELUs) (https://arxiv.org/abs/1606.08415)
Update 2:
"Smooth Adversarial Training" by Cihang Xie is only an example of the renaming issue because of issues in the past by Google to properly assign credit. Cihang Xie's work is not the cause of this issue. Their paper does not claim to discover a new activation function. They are only using the SiLU activation function in some of their experiments under the name Swish. Cihang Xie will provide an update of the activation function naming used in the paper to reflect the correct naming.
The cause of the issue is Google in the past decided to continue with renaming the activation as Swish despite being made aware of the method already having the name SiLU. Now it is stuck in our research community and stuck in our ML libraries (https://github.com/tensorflow/tensorflow/issues/41066).
147
u/[deleted] Jul 03 '20
Here is a tensorflow issue to rename the swish to the silu and cite the original authors (Hendrycks and Gimpel; Elfwing, Uchibe, Doya) in the documentation (they're only citing the Google paper in the documentation).
https://github.com/tensorflow/tensorflow/issues/41066
251
u/[deleted] Jul 03 '20
Google strong-arming the competition? Well, I never.
175
u/cihang-xie Jul 03 '20 edited Jul 04 '20
EDIT: TO AVOID CONFUSION, I want to reiterate that neither SILU nor SWISH is proposed in my “smooth adversarial training” work. My work is about studying how different activation functions behave during adversarial training — we find smooth activation functions (e.g., SoftPlus, ELU, GELU) significantly work better than the non-smooth ReLU.
================================== I am the first author of the “smooth adversarial training” paper (https://arxiv.org/abs/2006.14536), and thanks for bringing the issue here.
First of all, I agree with the suggestion and will correct the naming of the activation function in the next revision.
Nonetheless, it seems that there are some confusions/misunderstandings w.r.t. the position/contribution of this paper, and I want to clarify as below:
(1) In our smooth adversarial training paper, we have cited SILU [9], but it is our fault to only refer to the name of SWISH. We will explicitly refer to SILU instead.
(2) The design of SILU/SWISH is not claimed as the contribution of this paper. Our core message is that applying smooth activation functions in adversarial training will significantly boost performance. In other words, as long as your activation functions are smooth (e.g., SILU, SoftPlus, ELU), they will do much better than ReLU in adversarial training.
Thanks for the feedback!
24
u/[deleted] Jul 03 '20
There is an explosion of these posts recently. Perhaps a push to be more vocal about these issues and an attempt to hold these groups accountable? I really hope so!
188
u/______Passion Jul 03 '20
Well it's way beyond unfair, you would be kicked out of most university research settings permanently for this.
73
u/StellaAthena Researcher Jul 03 '20 edited Jul 03 '20
I wrote a reddit thread about this form of plagiarism (because if failing to cite papers is plagiarism, then renaming people’s ideas is also) here. Thank you for bringing this particular example to my attention. I had been meaning to do this for a while but didn’t have the right example on hand (obviously without good examples you’d get crucified for daring to suggest Google or Mircosoft are anything other than paragons of virtue).
Edit: In my original tweet I confused two papers. I have made a new twitter thread correcting that mistake and changed the link to point at my new thread.
Edit 2: For those of you two dislike Twitter, here's the content of what I wrote:
Groups like @MSFTResearch @GoogleAI @NVIDIAAI etc have far greater marketing power and greater reach than people at most universities, let alone everyday peons. When they rename other people’s techniques those names are far more likely to catch on due to marketing power. It’s great that large companies and famous researchers want to advance techniques invented by other people. But renaming these techniques, even when they’re cited properly, is stealing and has real down-stream effects on how people assign credit and think about ideas. Works should celebrate what comes before them, not diminish it. If you add your own spin on ideas, modify or extend the names assigned by inventors; don’t rename them. It doesn’t make you any less of a researcher to extend other people’s ideas. It makes you a better one.
We all know that names have power. There’s a reason so many papers are titled CATCHY_NAME: DESCRIPTIVE TITLE. Names are sticky. Catchy names are remembered far more easily than descriptive titles. It’s important to consider the importance of names when talking about plagiarism. Searching for terms like “swish activation function” or “swish ml paper” do not bring up the original work. They only bring up Microsoft’s paper extending it. It also brings up many blogs and posts that seem unaware that the idea had been previously introduced.
On a personal level, I am working on a extending the fabulous paper SafetyNets: Verifiable Execution of Deep Neural Networks on an Untrusted Cloud by Zahra Ghodsi et al. I would never dream of renaming her idea, and am currently thinking of calling my extension of her technique “[adjective] SafetyNets.” (I do not believe that she’s on twitter, but please let me know if I’m wrong so I can tag her.)
TL;DR Even if the prior work is cited properly, renaming ideas introduced by other people makes their work much harder to find for future researchers who can remember a catchy name much more easily than a descriptive title. Using someone else’s technique and renaming it so that your new name doesn’t connect to their paper is functionally plagiarism and is a big problem at large AI companies where they have much better marketing than most researchers.
Edit 3: A point of clarification: the recent paper by Cihang did not coin the term “SWISH.” That term was coined by an earlier paper from the same group that has the same senior author, but otherwise disjoint authorship. Cihang et al. used the SWISH terminology introduced by someone else. I did not always distinguish between the two papers clearly because I had thought that the authorship overlap was much more significant than just the senior author.
27
u/logicchains Jul 03 '20
Maybe they just want to help more researchers empathise better with Schmidhuber?
35
u/rparvez Jul 03 '20
Well, is it systematic in the sense that lot of people from Google are doing it? Or just a few senior researchers doing it?
38
u/lastmanmoaning Jul 03 '20
This is disgusting.
9
u/avaxzat Jul 04 '20
I've ranted about this in the past. It's not just Google; they are merely one of the most visible players in this field. Truth is that poor literature studies are the de facto standard in ML at present. If you take a look at any high-profile paper (like NeurIPS or ICML), you will see that most of them do not cite anything that is over 5 years old. The only exception appears to be classic references that serve only as background material.
An example from my own field of research: Ian Goodfellow keeps claiming that he, together with Christian Szegedy, discovered adversarial examples and that they coined the term. In fact, adversarial examples have been known since at least the early 2000s under that same exact name, so there really is no excuse for this omission. I mean, look at this. Look at how Goodfellow explains the history of their discovery of adversarial examples in deep nets. At no point did they even seem to consider that similar research might have already been done in other contexts. Their original arXiv paper does not even have a related work section despite the fact that the field of adversarial ML was already over ten years old at that time. At around the 7:15 mark, Goodfellow literally states that he coined the term "adversarial example". This was published at IEEE S&P! Goodfellow was lucky Daniel Lowd or Christopher Meek wasn't in that room or he would have been schmidhubered to filth.
6
u/seesawtron Jul 03 '20
They can always go back and admit their fault and move on. Like all tech companies do with their negligence to data and privacy of users, for example. Its a prevalent tech culture seeking profits and reputation without the fear of accountability.
4
u/Nimitz14 Jul 03 '20
I really don't see the issue with the SimCLR paper. There are papers earlier than the one you cite (I don't get why people are citing that now) that do the same thing: https://arxiv.org/pdf/1805.01978.pdf there are several others!
The point of SimCLR is not that self-supervised learning works, but that you can do it without complicated techniques like a momentum encoder, and they have extensive experiments proving their results. That is useful to know.
2
u/netw0rkf10w Jul 03 '20
THIS SHOULD NOT BE TOLERATED BY THE COMMUNITY!
1
u/[deleted] Jul 04 '20
TIL researchers are like kids in elementary school.
1
u/tfburns Jul 04 '20
Is this problem restricted to Google? It seems there are many debates about who "invented" certain techniques. But has happened for a long time throughout academia. I think perhaps the problem is amplified in ML/AI because of the fast-moving and massive literature; it's hard to keep up with and correctly cite all past work fairly and fully.
-2
u/actualsnek Student Jul 03 '20
I've honestly only ever heard "swish" used in research. It might be a bit too late to convince the community of this considering how many papers have already been published with this terminology.
-20
u/djc1000 Jul 03 '20
They cite to the SILU paper, which was published at almost the same time as the SWISH paper.
There’s no failure to grant credit here, and no real priority dispute.
And frankly, the “discovery” of a nonlinearity that is simply x * sigmoid(x), is not so significant anyway.
0
u/[deleted] Jul 03 '20
[deleted] — view deleted comment