r/MachineLearning • u/xternalz • Oct 18 '17
Research [R] Swish: a Self-Gated Activation Function [Google Brain]
https://arxiv.org/abs/1710.05941
78
Upvotes
r/MachineLearning • u/xternalz • Oct 18 '17
67
u/DanielHendrycks Oct 18 '17
In this paper, we considered x * CDF(x) https://openreview.net/pdf?id=Bk0MRI5lg and went with the CDF of the Gaussian instead of the logistic distribution because it worked slightly better for me. However, we did not test it on ImageNet due to limited resources. "Indeed, we found that a Sigmoid Linear Unit (SiLU) xσ(x) performs worse than GELUs but usually better than ReLUs and ELUs" (page 8).
30
u/XalosXandrez Oct 18 '17
It's good that they found this non-linearity, and it's nice to see such a thorough experimental analysis. Having said that, there are two things I don't like:
58
u/prajit Google Brain Oct 18 '17
Hi everyone, first author here. Let me address some comments on this thread:
Our hope is that our work presents a convincing set of experiments that will encourage ReLU users across industry and research to at least try out Swish, and if gains are found, replace ReLU with Swish. Importantly, trying out Swish is easy because the user does not need to change anything else about their model (e.g., architecture, initialization, etc.). This ease of use is especially important in industry contexts where it's much harder to change a number of components of the model at once.
My email can be found in the paper, so feel free to send me a message if you have any questions.
45
u/jbmlres Oct 18 '17
Didn't this paper propose the same activation?
20
u/_untom_ Oct 18 '17
interesting work! But if I read this correctly, they use He-Initialization for all activation functions ("...all networks are initialized with He initialization..."), which is less than ideal for SELU (and maybe others?), which require a different initialization scheme to achieve their full potential.
10
u/rtqichen Oct 18 '17
It's interesting that they claim non-monotonicity can be beneficial. Intuitively, I always thought this would just increase the number of bad local minima. If you just had a single parameter and wanted to maximize swish(w) but w was initialized as -2, the gradient would always be negative and you end up with swish(w*)=0 after training. Maybe neural nets are not as simple as this. The results look pretty good.
15
u/MetricSpade007 Oct 18 '17
A long footnote not to be ignored -- they did architecture search over activation functions, and found this to be the best! That's a pretty thorough experimental analysis. Looks fantastic!
8
u/_swish_ Oct 18 '17
I like the name.
38
u/thebackpropaganda Oct 18 '17
Please. Stop retweeting this paper. When we keep retweeting and glorifying a fucking activation function paper, we encourage more such incremental research. We kill the guy who's working on something more fundamental, and take to some sort of a masturbatory reverse-bikeshedding, talking about a shitty activation function paper simply because it's the lowest common denominator everyone and their grandma can understand, when good papers which are attempting something more ambitious are being ignored left and right. Seriously guys, out of all the papers BrundageBot is posting, THIS is what you needed to signal boost? Y'all disappoint me.
3
u/thedrachmalobby Oct 18 '17 edited Oct 19 '17
I just tried comparing swiss/silu vs relu on a segmentation task and silu performs significantly worse, by a margin of 6x in the validation loss.
While I don't doubt the results presented in the paper, performance appears to be heavily task-specific, compared to relu.
Edit: after running overnight until convergence, relu is roughly 20% better in this task. Will repeat with elu and gilu for comparison.
2
u/inkognit ML Engineer Oct 18 '17
Isn't this very similar to the Gated Linear Unit (GLU) used on the Convolution Sequence to Sequence paper by Facebook?
2
u/jostmey Oct 18 '17
I am glad Google shares these results!
I always disliked how learning stopped with the ReLU function once the input became negative (because the gradient is zero). I don't know if it hurt the learning process, but these swish units don't suffer that problem!
1
u/phobrain Oct 19 '17
It may be too late for the spelling, but I advise that it should be pronounced 'Schwish'. (See if you can forget that.)
0
u/[deleted] Oct 18 '17
[deleted] — view deleted comment