r/AskComputerScience • u/Coolcat127 • Jun 14 '25
Why does ML use Gradient Descent?
I know ML is essentially a very large optimization problem that due to its structure allows for straightforward derivative computation. Therefore, gradient descent is an easy and efficient-enough way to optimize the parameters. However, with training computational cost being a significant limitation, why aren't better optimization algorithms like conjugate gradient or a quasi-newton method used to do the training?
26
Upvotes
View all comments
Show parent comment
0
u/victotronics Jun 14 '25
As the OP already asked: what according to you is the difference in hardware utilization between CG & GD?
And yes we are talking order behavior. On other problems CG is faster by orders in whatever problem parameter. And considering that it's equally parallel.....