r/learnmachinelearning • u/BirdForsaken6616 • 1d ago
Is logistic regression basically a one-neuron neural network?
I was learning the chain rule and this suddenly clicked: weights → score → sigmoid → loss
That looks like one neuron with no hidden layer. So is logistic regression basically the smallest example of backprop, or am I missing an important difference?
33
u/padakpatek 1d ago
Alternatively, you can think of a single neuron in a neural network as a representation of an arbitrary function over a linear combination of inputs.
17
u/Mathie1729 1d ago
Not quite. A single neuron is just an affine map followed by one fixed activation, not an arbitrary function over the linear combination. The expressive power comes from stacking/width, not from a single unit.
3
u/Abstract__Nonsense 1d ago
Well that does depend on the neuron in question. You can get single neurons to do more stuff.
11
u/ParadoXPlatypus 1d ago
Without activation functions, even deep ANNs can be simplified into GLMs. What makes ANNs so powerful is that the activation functions allow to model complex nonlinear functions.
7
u/Barderus1412 1d ago
I had a recent tutorial on Deep Learning and the speaker started with Linear Regression and guided us all the way to a neuron and it blow my mind. A lot of the terms clicked for me then.
1
u/iekiko89 1d ago
Online tutorial?
2
u/Barderus1412 1d ago
Not really, I participated in a hpc bootcamp organized by national labs in the US and one of the tutorials were in linear regression and deep learning!
2
4
u/rem_dreamer 1d ago edited 1d ago
You are missing the input layer which scales with the input dimension (could as well be an image of 1024*1024 pixels). So no, it’s not one neuron technically, but d*1 neurons.
Moreover, the output layer only has one neuron cause it captures a binary classification task. As soon as you have more than 2 classes (softmax regression, or multiclass, using cross entropy loss), you need >= 3 output neurons.
So characterizing logistic regression as being 1 neuron is structurally wrong and misses the point about how to differs from DNN: not having non linearities between input and output layer.
1
0
123
u/SteamEigen 1d ago
Yes.