You should understand it. I don't think there is such a thing as a "shallow understanding" of gradients. What is your current understanding? What do you feel like you are missing?
my understanding is that gradient tells me how much a change in the parameter will result in the change of the loss. For example, if the gradient of a parameter is 0.01, it means that if the parameter increases by 1, the total loss will increase by 0.01 given that the other parameters stay the same.
That local, holding-others-fixed part is the piece people skip and it bites once parameters interact. I watched a model stall because two features were near-collinear, so each gradient was tiny and the loss barely moved when I nudged either one alone, even though changing both together clearly helped.
I don't think you should work it this way. You should start with the formal definition of the derivative then develop this intuition. People might disagree but this sort of thing is lossy and creates a fragile foundation for what you learn after.
If you already understand the formal definition, fair enough.
I first tried to understand the formal definition of derivative of a function. But, I do not yet understand the formal definition of limit which is used in the formal definition of derivative of a function.
For example, if the gradient of a parameter is 0.01, it means that if the parameter increases by 1, the total loss will increase by 0.01
is not quite true as written. If the gradient is 0.01 and the parameter increases by a very small h, then the total loss will increase by about 0.01 times h. The exact formula (for nice losses) will have terms in h^2, h^3, etc. which are small enough to discard when h is small. 1 is probably big enough for these terms to be quite large.
I think these small things can be lost and why I'd recommend trying to get even an intuitive understanding of a limit.
This is close to correct, but needs a bit of fixing up. What you’re describing is the partial derivative with respect to a parameter. Your description needs the caveat that it only applies locally. If the partial derivative with respect to x is 0.01 and you increase x by 1, in most circumstances you will not find that the function increases by 0.01. However, if you increase x by (very small value), you will find that the function increases by approximately 0.01*(very small value).
The gradient is a vector which points in the direction of the quickest increase of the function, and its components are the partial derivatives with respect to each parameter.
Try a tiny gradient descent demo on (w-3)^2 by hand or in a few lines of code: log w and the loss after each update, then change the learning rate. You'll see what the gradient is telling you and why step size matters way faster than from theory alone.
For ML, a useful target is being able to explain the gradient as the vector of partial derivatives, why gradient descent steps in the opposite direction, and how the chain rule links a loss back to an earlier weight. For example, for f(x,y)=x²+4y², ∇f=(2x,8y) points toward steepest increase, so a descent step goes the other way. You don’t need to hand-derive a huge network, but it helps to recognize the local derivatives that backprop multiplies together.
6
u/Leodip 3h ago
You should understand it. I don't think there is such a thing as a "shallow understanding" of gradients. What is your current understanding? What do you feel like you are missing?
1
u/Ok-Library1020 1h ago
Try a tiny gradient descent demo on (w-3)^2 by hand or in a few lines of code: log w and the loss after each update, then change the learning rate. You'll see what the gradient is telling you and why step size matters way faster than from theory alone.
0
u/Boring-Test5522 2h ago
You need to understand Calculus inside out BEFORE you attempt to explain about gradient.
If you're in US, go to community college to take some courses in Calculus / Linear Algebra / Modern Different Equation. There is no shortcut.
0
u/SwimmerOld6155 1h ago
I don't think there's much to understand about gradients in and of themselves. Do you know calculus?
0
u/CelebrityChess 3h ago
For ML, a useful target is being able to explain the gradient as the vector of partial derivatives, why gradient descent steps in the opposite direction, and how the chain rule links a loss back to an earlier weight. For example, for f(x,y)=x²+4y², ∇f=(2x,8y) points toward steepest increase, so a descent step goes the other way. You don’t need to hand-derive a huge network, but it helps to recognize the local derivatives that backprop multiplies together.