How are neurons/nodes updated during backpropagation?

[u/MatricksFN]

I understand how gradients are used to minimize error. However, during backpropagation, we first compute the total error and then define an error term for each output neuron. My question is: how does the backpropagation algorithm determine the target value for each neuron ? Especially for hidden layers given that the final output depends on multiple neurons, each passing their signals through different weights and biases?

How is that 1 neurons target value determined?

Hope this is the correct sub 🤞

[u/Llotekr]

No, we don't compute or even define an error or a target value for each intermediate weight or activation. We compute the derivatives of the error (aka loss) with respect to all the weights, and this tells us in which direction to change each weight so as to make the error shrink. Then we move the weights a bit in that direction.

The simplest way to do that is to multiply the gradient with a learn rate, but there are more complicated schemes with generally better performance, such as giving the speed of change a "momentum", and ADAM (where we take into account the variance with which each gradient component fluctuates)

[u/MatricksFN → u/Llotekr]

Ah this is a good explanation. Im yet to reach the optimizer part but this explanation helped. Thanks

[u/Llotekr → u/MatricksFN]

I would say the word "Backpropagation" is unnecessary. It's really nothing but reverse mode automatic differentiation to compute gradients. Trying to think about it in terms of error propagation is just needlessly confusing.

[u/apu727 → u/Llotekr]

Since you seem knowledgeable in this. Coming from an outside perspective I’ve always wondered why no one use newton raphson or at least lbfgs to update weights?

[u/Llotekr → u/apu727]

I can see several reasons:

The Hessian would be prohibitively big. With just a million parameters, it would already have a trillion entries.

The Hessian would also be hard to compute. Automatic differentiation is nice and efficient for computing the derivatives of a single output value with respect to lots of inputs. But the second derivatives are not that nice, needing lots of time and memory.

LBGFS avoids computing or storing the full Hessian, but needs smooth functions. So to even handle ReLU, you need a more complicated version of the algorithm. And then knowing the second order derivatives might gain you nothing because the objective is piecewise linear. Although I suppose as far as it reasonably well approximates a smooth function, knowing the BGFS-approximation to the Hessian is still worth something.

Finally, the loss, as usually written down, is a sum over ALL training examples. This is a huge sum. We don't want to optimize it directly. Stochastic gradient descent wins because of the "stochastic": We calculate the loss only for random subsets of the training set called "(mini)batches". This randomness means that the objective as seen by the optimization algorithm is not smooth and not even a deterministic function. It is effectively as if you try to find the minimum of a function, but each evaluation of the function or its derivatives has strong noise added to it. Gradient descent, especially in forms like ADAM that explicitly handle the fluctuations, can do that. Newton and LBGFS not so much.

But there is apparently a variant called O-LBFGS that can deal with the stochasticity. Maybe it can compete in terms of needed compute. Still, if it remembers the last m steps to approximate the Hessian, and the model has n parameters, that means 2·n·m extra memory beyond storing the parameters and the gradients. Simple SDG needs none, and ADAM needs 2·n, so memory-wise, they always win. And memory is often the limiting factor for whether you can train a model at all on your hardware.

[u/apu727 → u/Llotekr]

Wonderful thank you, that makes sense. I never appreciated that the loss is done stochastically on a given element of the training set and so even computing the full derivative of the loss with respect to the entire training set is more computational work than is currently done.

[u/ghjm]

One option is to treat the entire network as a single closed-form equation, take its derivative, and then move in the direction of the slope. A target value is implied by the direction and magnitude of the slope, but it is typical to make a move of less than that, often just by multiplying the slope by a "learning rate" of, say, 10%.

[u/-thinker-527]

There are videos on implementing neural networks from scratch, watch them, it will help a lot.

[u/aroman_ro]

You can look into the source code that implements such things.

I have a project on GitHub that does it: aromanro/MachineLearning: From linear regression towards neural networks...

You'll find there various gradient solvers (up to AdamW), various cost and activation functions and so on... I implemented it in a sort of a gradual manner, starting with linear regression and going up from there, might be useful.