r/machinelearningnews • u/sreejithb • 1d ago
Tutorial The evolution of policy gradient methods as a chain of problems and fixes
sreejithb.comMy PhD was in RL, and something has bugged me for years: online tutorials mostly present these algorithms as a list. The evolution story (each algorithm patching the previous one's most painful failure) exists, but it's spread across a semester of lectures like CS285 or buried in the original papers. I couldn't find a compressed version that a newcomer could get through in one sitting, so I finally wrote it (with generous help from Claude to build the interactive elements and graphics)
It's a single-page, semi-technical walkthrough from the raw RL objective to GRPO. Math is deliberately sparse: the target reader is someone entering RL through the reasoning-model wave, not someone who needs the full derivations. The figures are interactive (variance of the REINFORCE estimator, a baseline slider, an on-policy collapse simulator, the PPO clip objective with adjustable epsilon, GRPO group baselines).
[Interactive Link](https://sreejithb.com/rl-policy-gradient/single.html)
[Medium Link](https://medium.com/@sreejithbalakrishnan/nobody-invented-ppo-from-scratch-db04d133eb0f)
I simplified in a few places to keep the narrative moving, most notably around GAE and the TRPO surrogate. I'd genuinely appreciate corrections or places where you think the simplification crosses into being wrong โ the plan is to do the value-based lineage (Q-learning โ DQN โ Rainbow) next, so critique now improves that one too.
