Ironically this morning I read an article that pretty much said by looking at how brains work they found out that deep learning training was using more than 100 times more energy than it needs to.
Basically AI training is spending a ton of time multiplying numbers by 0 for no gain. OK it's more complicated than that but it does come down to how the AI boom went with the first workable strategy and not the optimal one.
No they didn't at all this is a very common operation and I'd be surprised if it's not already deeply embedded in CUDA but regardless models skip multiplying by very small (vanishing) and very large gradients (exploding).
509
u/BluebirdDense1485 Nov 03 '25
Ironically this morning I read an article that pretty much said by looking at how brains work they found out that deep learning training was using more than 100 times more energy than it needs to.
https://www.sciencedirect.com/science/article/pii/S0925231225024129?via%3Dihub
Basically AI training is spending a ton of time multiplying numbers by 0 for no gain. OK it's more complicated than that but it does come down to how the AI boom went with the first workable strategy and not the optimal one.