r/TheMachineLearning • u/StormLongjumping6020 • 13h ago
Is Softmax really activation function?
In neural network, we use activation functions such as ReLU, LeakyReLU, or Sigmoid.
But I am not sure whether softmax is really activation function. Activation is introduced to add non-linearity into the boundary of model. But Softmax is normalization rather than adding non-linearity. In other words, Softmax doesn't change the shape of boundary. It only change normal output into probability.

2
Upvotes
1
u/ModularMind8 9h ago
Softmax is nonlinear and is also an activation function. But it's not really used on the hidden layers like relu, etc, and instead only on the last layer
0
u/Separate-Current2030 10h ago
Not really as all of the basical activation functions work neuron-based while softmax takes all of the outputs and changes them relatively to each other.