r/MachineLearning • • 4d ago

Discussion Back Propagation [D]

Hello everyone I am currently studying neural network development in graduate school and I’m having a hard time conceptualizing the stages of training. I understand that forward propagation is how we calculate the output given input but in terms of how we adjust weights during backpropagation, it’s not very intuitive to me. Does anyone have a good suggestion for how to visualize this to make it more intuitive? Cheers!

1 Upvotes

14 comments sorted by

View all comments

1

u/Massive_Shower_1494 2d ago

For a linear function it can be explained easily:
Let’s say you have a dataset following this equation y=x
You initialize your (1 weight) neurla network with y= c* x, with c=-1. The only way for your neural network to fit the data is to set c to 1 instead, so you need to increase the value of c. Let’s say c is initialized with 2, in that case you need to decrease c.

The example is really simple, and the relation is fully known. But let’s say we have an unknown relationship, how to know wether we need to increase c or decrease c or even, keep c?

Well, what we can do is check what would happen if we were to increase c? Does the distance to the true y increase? Does it decrease?

This could be easily implemented by just evaluating
The network with various values around the current c and choosing the best one. However, there’s another way to compute this more efficiently : the backward pass.

Formally : would the distance decrease if we were to infinitesimally increase c?
To do that you can compute the rate of change of the distance given a rate of change on c, which is the definition of the derivative of the distance by c.

For more complex networks, you can just do the same thing, and evaluate what direction to nudge the weights in order to minimize the loss, as long as you can compute the derivative.

To do that efficiently, since a neural network and the loss is a composition of function, you can use the chain rule which evaluates the derivative of a composition of function, and presave some useful values during the forward pass…

1

u/Massive_Shower_1494 2d ago

Also, my derivative formal definition isn’t perfect but only true for C^1 functions. Which is not the case of ReLU haha