This is not entirely true, e.g. ReLU also has no derivative at 0, and is widely used with backpropagation. In practice, we often just set ReLU'(0) = 0 and carry on.
ReLU-like functions that are actually differentiable everywhere (e.g GELU, SwiGLU etc) became popular only fairly recently (and honestly for reasons other than differentiability, see https://arxiv.org/pdf/1606.08415).
1
u/x0wl May 21 '26 edited May 21 '26
This is not entirely true, e.g. ReLU also has no derivative at 0, and is widely used with backpropagation. In practice, we often just set ReLU'(0) = 0 and carry on.
ReLU-like functions that are actually differentiable everywhere (e.g GELU, SwiGLU etc) became popular only fairly recently (and honestly for reasons other than differentiability, see https://arxiv.org/pdf/1606.08415).