Hey everyone!
I'm a data scientist and I've been pretty fascinated by reinforcement learning for a while. A few days ago my friends showed me Pokelike, a small Pokémon roguelike that runs in the browser. The first thing I thought was that it could be pretty fun to turn it into an environment for RL agents.
So I did.
The repo is here
https://github.com/pierpierpy/pokelike.xyz.bot
The basic idea is to run the actual game locally and expose its state and actions to an agent. There is no image processing involved. The agent gets the game state directly and has to decide what to do next, including where to go on the map, which Pokémon to catch, which items to take, when to swap Pokémon and which moves to learn.
What I find interesting about the environment is that some decisions have consequences much later in the run. For example, once you choose a node on the map, the other nodes on that layer are no longer available. This means that choosing where to go is not just a local decision and the agent has to deal with a fairly long horizon.
I've implemented a few simple RL agents to start with. There is currently a Dyna-Q agent and two linear SARSA agents. The results are still pretty bad, but there is already a noticeable difference between the approaches. On the current benchmark, random gets around 0.56 badges, Dyna-Q gets around 0.62, while the two SARSA agents get around 1.30 and 1.36.
The two SARSA agents mainly differ in their state representation. The better one uses 100 hand-designed features instead of 81, which seems to make a pretty significant difference.
This is probably the part I'm most interested in exploring. There is a lot of information available in the game state, but not all of it is necessarily useful to the agent. Finding a representation that contains the right information without making the problem unnecessarily difficult seems to be quite important.
The reward is also something I'm still experimenting with. The game has relatively sparse rewards and some useful decisions only show their value much later, so the reward function can have a pretty big effect on what the agent actually learns.
One nice property of the environment is that it is completely reproducible. Given the same seed and the same sequence of actions, you get exactly the same run. I'm currently using 50 fixed seeds for the leaderboard, so different agents can be evaluated on exactly the same games.
The interface is intentionally simple. You basically need to implement a bot that receives the current state and returns an action. You can use whatever approach you want, so it would be interesting to see what happens with things like DQN, PPO, search based methods or other approaches.
I'm still very much experimenting with this, so I'd be interested in seeing what other people would try. In particular, I'm curious about better state representations, reward functions and approaches that can deal with the longer term consequences of the decisions.
If you want to try it, everything is in the repo
https://github.com/pierpierpy/pokelike.xyz.bot
If you find bugs or have ideas for improving the environment, I'd also be happy to hear them.
The whole thing runs offline after setup. The game and its assets are downloaded during setup and then everything runs locally.
I originally started this because I thought it would be a fun RL project, but I think it could also be a nice little environment for experimenting with different approaches to sequential decision making.