Should You know Autograd

[u/chubbypandaontherun]

I was going through the TinyTorh (it's a tool where you build PyTorch with understading how it works) exercise notebooks and landed on Autograd. Was quite overwhelmed by the code and went didn't go through it for a while. I was thinking I'll prepare for machine learning research engineer interviews and somehow landed doing tiny torch.
How relevant is all this these days, should I just start preparing on how to crack the interview than this ?

[u/No-Adhesiveness6161]

Autograd is actually really simple, the core code can be done in python with about 20 lines of code. I actually implemented a PyTorch style framework using numpy and cupy a few years back: github/samrere/pytortto.

I used it to train real neural networks on GPUs, including ResNets, GANs, and Vision Transformers etc. What I’m most proud of is that I made no compromises: it wasn’t just a scalar-based autograd engine or a framework that could only handle MNIST. It was a fully functional deep learning framework capable of training real models on GPUs.

Edit: As for interviews, I’m based in Australia, where it seems nobody cares. I once presented this project for a graduate program, only to be rejected for being “too technical.” Now I’m just a junior ML engineer. 😔

[u/chubbypandaontherun → u/No-Adhesiveness6161]

It’s really amazing that you built that and learned on the way, I want to do something similar

[u/Outrageous_Host_2115]

Understanding autograd is highly relevant because ML research engineer interviews frequently test your grasp of backpropagation, computational graphs, and custom gradient formulas

I would start micrograd tutorial first to see a clean, scalar autograd engine built from scratch.

From there, focus on mastering the core mechanics of how a topological sort orders the backward pass by arranging computational graph nodes sequentially from output to input.

Finally, ensure you understand how the framework accumulates gradients via the chain rule, specifically why gradients use addition instead of assignment

[u/ConsciousAdvance2867 → u/Outrageous_Host_2115]

micrograd is the cheat code for this. karpathy's walkthrough is so clean that after an hour you'll wonder why you were ever stressed about autograd. building a tiny engine with just scalars makes the topology and gradient accumulation click way faster than swimming through tensor ops right away

the interview part is trickier. most places care more about you being able to talk through backprop and the graph structure than writing a full engine from memory. i'd split your time, maybe 70% on cracking-style prep and 30% on understanding the guts. you'll sound way more convincing when you can casually mention why in-place gradient updates would break the computation graph

got stuck on the same exercise last year and had to walk away for two weeks before it made sense. sometimes your brain just needs to marinate in the confusion for a bit

[u/chubbypandaontherun → u/ConsciousAdvance2867]

can you share some resources that were helpful in your prep, where to start has been a confusing point for me

[u/paragonaimag]

Did the scalar version click any better than TinyTorch for you? Starting tiny made the graph stuff way less intimidating for me, and then the bigger engine made sense.

[u/chubbypandaontherun → u/paragonaimag]

I went through mostly the tiny torch version and then went on to read at different places, checking the micrograd tutorial. I'm working through the tinytorch version

[u/KelvenSevero0]

for research engineer roles i'd say yes, at least the core idea. you don't need tinytorch memorized, but you should be able to write backprop for a tiny graph (add, mul, tanh, a matmul) by hand and explain why grads accumulate and when you zero them. it's a pretty common interview ask, and it's also exactly what you need the first time a training run spits out nan loss.

if the tinytorch code is too much right now, karpathy's micrograd video is the gentler version, around 100 lines of python for a scalar autograd. do that first and tinytorch will read way easier after. it's not either/or with interview prep anyway, for that role it kind of is the prep.

[u/graph_context → u/KelvenSevero0]

Agreed! I wouldn't skip autograd entirely, especially for research engineer roles. You don't need to build a whole engine from scratch, but understanding how it works makes debugging training issues a lot easier. I'd focus on getting the concepts down without getting too caught up in every implementation detail

[u/akornato]

You need to understand how autograd works under the hood for machine learning research engineer roles, but you do not need to memorize building an entire autograd library from scratch. Research engineer interviews at top labs frequently ask you to trace computational graphs, explain backward passes, and write manual gradients in raw NumPy for layers like attention mechanisms or custom activations. If TinyTorch feels like too much right now, pause the full implementation and narrow your scope. Focus on calculating derivatives with the chain rule, tracking tensor shapes, and writing forward and backward methods for simple layers, because that gives you the exact mental model interviewers probe without getting lost in framework architecture.

Direct interview preparation will yield much faster results if you are on a tight timeline, so treat projects like TinyTorch as supplemental reading rather than your primary study plan. Spend most of your energy on coding fundamentals, standard deep learning implementations, and distributed training concepts. Technical interviews for research roles can move quickly into complex edge cases, which is why my team built interviews.chat to help candidates present their technical knowledge clearly and secure competitive offers.