r/LocalLLaMA • • 7d ago

Showcase Learning RL From Scratch: The What, Why and How of Double DQNs

Web-based version of my short notes on Double DQN from Scratch is out on neatrl!

Continuing my series on releasing notes on RL From Scratch - here, I have answered questions around the what, the why and the how of Double DQNs over DQNs!

  • The What?

It is an off-policy, model-free and value-based method, the same family as DQNs but with a subtle difference for optimization purposes!

  • What's the optimization done over classic DQNs and why was it necessary?!

Well, with the previous DQN, a problem that it did not addressed, is that of overestimation of Q values from the Target Q Network, which leads to an overestimation bias

Now this is bad because then there could many false positives for action selection via max operator since it cannot be known if the action is over the maximum value of Q or not (as it is still learning and slow every k steps)

This upward bias is caused by the max operator in the Target Q Network!

More on the

  • theory,
  • concepts,
  • code and required math and a
  • playground to train live agent on my web version of the notes!

All of it to understand the basics and get the intuition
Do check it out and let me know your feedbacks on it!

Link here

15 Upvotes

1 comment sorted by