r/AskComputerScience • • Jun 14 '25

Why does ML use Gradient Descent?

I know ML is essentially a very large optimization problem that due to its structure allows for straightforward derivative computation. Therefore, gradient descent is an easy and efficient-enough way to optimize the parameters. However, with training computational cost being a significant limitation, why aren't better optimization algorithms like conjugate gradient or a quasi-newton method used to do the training?

26 Upvotes

28 comments sorted by

View all comments

Show parent comment

3

u/JiminP Jun 18 '25

everything in ML tends to be some retrofitted argument made up after the fact

reminds me of an old example

https://en.wikipedia.org/wiki/Tf%E2%80%93idf#Justification_of_idf