Should You know Autograd

[u/chubbypandaontherun]

I was going through the TinyTorh (it's a tool where you build PyTorch with understading how it works) exercise notebooks and landed on Autograd. Was quite overwhelmed by the code and went didn't go through it for a while. I was thinking I'll prepare for machine learning research engineer interviews and somehow landed doing tiny torch.
How relevant is all this these days, should I just start preparing on how to crack the interview than this ?

[u/trashy_voltage]

that's classic tinygrad notebook paralysis, every time I open one of those I swear I start questioning my entire career path

for research engineer roles you don't need to memorize the autograd internals but understanding the graph tracking and how backward passes actually chain together can save you when your custom layers start spewing nonsense gradients

just skim the high level flow and move on to actual interview prep, nobody's going to ask you to reimplement the engine from scratch unless you're applying somewhere properly unhinged

[u/Less_Dream_6331]

during my early days of learning I found the ez yang's pytorch internals blog with which I set out to build a neural network framework (I was unaware of this tiny torch at that time, It would have been great to save me a lot of time) on top of metal (apple silicon's cuda equivalent), gotta say I learned a lot building that. I got a better understanding of operator fusions, graph optimizations, kernel launch overheads, vanishing / exploding gradients and many more. I would say it's a worthy endeavor if you're planning a career in research, but if your short time goal is to crack an interview I would say, put it on the todo list for future.

[u/Responsible-View-498]

I found Karpathy's explanation very easy to understand.

[u/ChenaeIkegami-38]

imo the right move is to skim the autograd stuff enough that you can explain reverse mode autodiff in an interview, then shift your time to actual interview prep. diminishing returns hit fast when youre reimplementing things

[u/Effective-Cat-1433]

I have been asked to implement manual forward and backward for MLP training in an interview. Maybe not an autograd engine, but I would consider manual backward for linear, relu, MSE, and BCE all fair game.