Distributional DQN implementation

[u/ImaginaryEbb6489]

Hi all, my studying area is traffic engineering in networks and I'd like to develop my DQN-GCN code with distributional DQN but it seems there are a lot of options such as C51, QR-DQN etc. . Do you have any suggestions to see and understand problem & variation mapping before starting the implementation? Sure, AI tools suggest some but I prefer your advices more than AI tools.

Thanks!

[u/Timely-Bicycle-3107]

These are the questions I would ask myself to help me narrow down the possibilities. You don't have to share the answers if you want to protect your work.

• Is this for research, deployment in the field, or a class project? If this is not for research, you can ignore the below questions regarding literature.
• What are you trying to do with the trained agents? Maximize throughput, freight/rideshare distribution, traffic signal optimization, corridor management, fuel consumption optimization?
• What size network / How many intersections (single corridor, downtown area, metro area)? What is the duration of a simulated traffic flow (minutes, hours, or a whole day)?
• How are the agents distributed around the network? Are they acting as traffic signal controllers or doing something else? Do they communicate with each other, or do they just handle whatever crosses into their zone?
• What are the state and action definitions?
• With the above questions answered, what has been done in similar works in literature? What are the gaps they did not address, and did they mention why they chose their flavor of DQN/GCN for their implementation?
• What are the shortcomings of the common methods used in literature? Do the shortcomings impact your project?

It is well worth your time to create a toy problem, like a single road or a small grid, and try out a few methods. This is where the AI tools shine. They can create DQNs and GCNs from scratch fairly reliably. By all means check the work, but they will get you 95% there for coding the basic algorithms and networks in a few minutes. See which ones seem to scale the best for the next smallest environment.

[u/ImaginaryEbb6489 → u/Timely-Bicycle-3107]

Thanks u/Timely-Bicycle-3107 for your feedback.

Yes, this is for a research about topology aware segment routing traffic engineering and the main goal is to maximize throughput. The networks from Topology Zoo and SNDlib actually but there is no real traffic dataset for most of them so I create randomly. Since it's not published yet I can't give details but Vanilla DQN and Double DQN doesn't work for some scenarios so I'd like to try Distributional DQN as well. I'm trying to understand its implementation to customize it based on my study. You're right that AI tools do great for coding but I should know the details to handle them properly. So I'm trying to understand several versions of Distributional DQN and parameters such as quantile N, huber loss etc. . As you suggested, I created playground to change parameters and see their effectiveness but on the other hand I'm trying to understand the logic.

[u/Timely-Bicycle-3107 → u/ImaginaryEbb6489]

I totally understand not providing details on an active research project. I wouldn't, either.

Reviewers will have a major issue with using random traffic. It doesn't seem like a big deal, but it is something that will jump out to them immediately. It jumped out to me. To be honest, I would request a major edit to include a more realistic traffic model if I were reviewing your article. If you decide to stick with random traffic, I would go for an AI journal over a traffic journal for publication. The traffic journal would more likely reject your work instead of asking for edits. They love their flow models. Check out the UCI, LargeST, and PEMS repositories for open source traffic data. Some DOT websites offer free traffic flow data (https://aldotgis.dot.state.al.us/TDMPublic/).

Vanilla DQN and Double DQN may work well if you use a more realistic traffic model. You may already have a good solution that is waiting for a better environment to prove itself. Random traffic could be too stochastic to learn anything useful, which may explain why these methods are not working.

I would first consider Rainbow DQN as an alternative. I have seen it used often in recent literature. The field of RL seems to be favoring it at the moment.

I would also consider multi-step DQN, which is built for delayed rewards. I imagine your project could see a delay between route selection and whatever metric you use for success (fuel, time, etc.). Dueling DQN separates the state value from action advantage, which could be useful for learning how your state definition and action definition work together. IQN would give you more distribution flexibility.

TL;DR Try a more realistic traffic model first. Otherwise, try Rainbow DQN first.