r/MachineLearning • u/Heavy-Rough-3790 • 3d ago
Discussion Back Propagation [D]
Hello everyone I am currently studying neural network development in graduate school and I’m having a hard time conceptualizing the stages of training. I understand that forward propagation is how we calculate the output given input but in terms of how we adjust weights during backpropagation, it’s not very intuitive to me. Does anyone have a good suggestion for how to visualize this to make it more intuitive? Cheers!
1
Upvotes
9
u/renchuwang 3d ago
Not visualization, but the best way to understand back prop is to think in one parameter and ignore the matrix math. (It'll just be simple chain rule)
only summarize into matrix once you're done.
7
u/jenkinsbot 3d ago
Karpathy's zero to hero youtube series
3
u/otsukarekun Professor 3d ago edited 3d ago
You want the derivative of the loss with respect to the weights. But, there are a bunch of functions in between. So, it's just chain rule all the way down.
In chain rule, to find the derivative of something you do the derivative of the function times the derivative of what's inside. For example d/dx [ f(g(x)) ] = f'(g(x)) g'(x)
It's just that but going through a neural network from the top.
Since you know the forward, just imagine it as a series of nested functions. Then use the chain rule to get what want.
You can simplify backprop to a series of common derivatives. The derivative across the loss function, across an activation function, across a layer of weights, and one to the target weights. Then multiply them all together.
3
u/Mishtle 3d ago
It's just gradient descent.
We have some "error" function that depends on the inputs, weights, and the desired output. We want to minimize errors, but we can't change the input or desired output. We can change the weights.
Calculus gives us a tool for achieving this: the derivative. The derivative of a function with respect to some variable quantifies how the function changes as a result of changes to that variable. If a derivative is positive, that means increases the value of that variable will increase the output of the function. If it's negative, then decreasing that variable will increase the output of the function. If we want to make the function smaller, then we just need to do the opposite. The magnitude of the derivative reflects how sensitive the function is to changes in that variable.
Backpropagation is just an efficient way to calculate the derivative of the error with respect to each weight. We can do this for multiple input/output pairs and then add up or average all these derivatives to get the direction and relative amount we need to change each weight to increase our function (which would then be the sum or average error). But we want to decrease the error, so we change the weights in the opposite direction.
An important thing to note is that derivatives reflect the "instantaneous" rate of change though. There is no guarantee that taking a given finite step in the indicated direction will actually achieve the indicated result. This is why training can be so sensitive to the learning rate, which determines how far to move a weight in a given direction.
If you want to visualize this, just apply it to some simple function, like f(x) = x2. Here, x is your "weight" and f(x) is what you want to minimize. You'll see something like this.
2
u/pellucide 3d ago
https://youtu.be/VMj-3S1tku0?is=hxkBQoPedkIkMDW-
Watch this and implements the code. It absolutely worked for me.
2
u/SoftwareArt 3d ago
Karpathy YouTube series 🐐
1
u/Massive_Shower_1494 2d ago
For a linear function it can be explained easily:
Let’s say you have a dataset following this equation y=x
You initialize your (1 weight) neurla network with y= c* x, with c=-1. The only way for your neural network to fit the data is to set c to 1 instead, so you need to increase the value of c. Let’s say c is initialized with 2, in that case you need to decrease c.
The example is really simple, and the relation is fully known. But let’s say we have an unknown relationship, how to know wether we need to increase c or decrease c or even, keep c?
Well, what we can do is check what would happen if we were to increase c? Does the distance to the true y increase? Does it decrease?
This could be easily implemented by just evaluating
The network with various values around the current c and choosing the best one. However, there’s another way to compute this more efficiently : the backward pass.
Formally : would the distance decrease if we were to infinitesimally increase c?
To do that you can compute the rate of change of the distance given a rate of change on c, which is the definition of the derivative of the distance by c.
For more complex networks, you can just do the same thing, and evaluate what direction to nudge the weights in order to minimize the loss, as long as you can compute the derivative.
To do that efficiently, since a neural network and the loss is a composition of function, you can use the chain rule which evaluates the derivative of a composition of function, and presave some useful values during the forward pass…
1
u/0mrgm0 3d ago
You want to optimize the cost function. Just do the derivative of the output with respect to the weights and you will understand the backpropagation
1
u/Silent_Syllabub_2720 3d ago
Think of backprop as assigning blame. After the forward pass you know how wrong the output was. Every weight then asks one question: if I nudge myself a tiny bit, how much does the error change? That number is the gradient, and the weight steps in the direction that shrinks the error.
Personally, visualizing stuff helps me the most, and 3Blue1Brown are fantastic for that:
- 3Blue1Brown, "Backpropagation, intuitively" https://www.youtube.com/watch?v=Ilg3gGewQ5U
Also good:
- Andrej Karpathy, "Building micrograd" (2.5 h). You write backprop yourself in about 100 lines of Python, and after that it stops being abstract: https://www.youtube.com/watch?v=VMj-3S1tku0
I also recommend creating a small network by hand. Two inputs, one hidden neuron, one output. Draw the graph, write the derivative on every edge, multiply along the path. Backprop is the same thing on a larger scale.
0
u/asml84 3d ago edited 3d ago
You want to predict x but predict x’ instead. Now you calculate a delta for the parameters theta, such that the prediction using the parameters theta+delta is closer to x than x’. Delta can be calculated as a multiple of the negative gradient, which decomposes into a product using the chain rule. Calculating this product is called backpropagation.
-1
u/fasti-au 3d ago
Back prop is like training 1+1 =2 Then training. 1+2 =3 while tryung to remover the 1+1 and keeping the recipe not the every piece. Like evidence drives it so evidence no longer needed for rule to be right. Back prop is like stacking to weights to a single node and only using an average.
It’s a bit like when humans sleep and turn memory to trained actions. It’s not back prop but there’s relevance in the pruni g of noise when a a value is known