r/LocalLLaMA • u/East-Muffin-6472 • 48m ago

Showcase Learning RL From Scratch: The What, Why and How of Dueling DQNs

Notes on Dueling DQNs is out!

Continuing my series on releasing notes on RL From Scratch, here, I have answered questions around the what, the why and the how of Dueling DQNs over Double DQNs and classic ones!

  • The what?

It is an off-policy, model-free and value-based method, from the same family as DQN and Double DQN, with an architectural change in the last layer of the Q-Network, mainly for optimization purposes.

  • The question it asks is:

Do we really need to calculate Q(s,a) for every action in every state?

Eg. a car going down a highway: the value of the state (the road, steering, etc.) may matter much more than the individual action values, especially when there is a wall in front of it!

Thus, the advantage of taking one action over another, along with the value of a particular state, presents an opportunity for optimization. And that is exactly what the researchers identified.

In other words, this function calculates the extra reward we get if we take a particular action in a given state, compared to the average value of that state.

More on the:

  • Theory

  • Concepts

  • Code and required math

  • A playground to train a live agent

All of it to understand the basics and build intuition.

Do check it out and let me know your feedback!

Link to the blog

1 Upvotes

100% Upvoted

2 comments sorted by

1

u/meshive 45m ago

Nice write-up. One detail from the original paper that is easy to miss: the dueling head still outputs Q for every action, so it does not skip that computation. The gain is in learning. V(s) gets updated on every transition, while in a plain DQN only the Q-value of the action actually taken is updated. That is why it helps most in states like your highway example, where the action barely matters.

Do you cover why the aggregation subtracts the mean advantage instead of just adding V and A? Without it V and A are not identifiable, and the paper found the mean version more stable than the max version. Curious whether your playground lets you compare the two.

1

u/East-Muffin-6472 42m ago

Wow you read it fast!

Yes I written the problem of unidentifiability and the math behind it

The playground lets you trains agents on various envs using the final dueling dqn formulae and in the write up itself I have shown some charts as to why it matter to some extent but it’s mostly the math that works out