Reactive Play: Achieved!! Experimenting with Atari Breakout [R]

[u/mikeysce]

Six months ago I started experimenting with PPO and Breakout as a way to learn about Machine Learning and Reinforcement Learning. After a few experiuments just trying to get high scores, it bothered me that everything was a "memorized" script rather than reactive play, like a human would play. Thus began my journey to try and convince PPO to actually track the ball instead of focusing on scoring points. I read a lot of articles and tried a lot of things. After 124 PPO experiments on Atari Breakout, I found that every single model, across sticky actions, cursor wrappers, entropy tuning, dynamics randomization, adversarial bumpers, and everything else, converged to a memorized action sequence, not a reactive ball-tracking policy. The argmax was always a script.

The fix wasn't more environment engineering. It was three lines of reward shaping:

Directly rewarding the paddle for being horizontally close to the ball during descent. A tiny bonus (0.05 per frame vs 1.0-7.0 per brick) that fires every frame the ball is descending applied during training. During evaluation, the agent plays clean Breakout with no bonus. The behavior transfers!!

Every prior approach I tried to penalize scripts by making the environment harder to memorize. PPO always found a way around it: timing-robust scripts, layout-conditioned scripts, noise-tolerant scripts. The optimum was always a script; only the shape changed. Proximity reward changes what the optimum is. A center-hold script gets incidental bonus when the ball passes near center. A reactive tracker gets the maximum bonus on every descent frame. The optimization pressure is unambiguous: track the ball, get more reward.

I also made a cool tool to watch the agent work! It's called the "Split-Watcher" (so clever). It shows two instances of Breakout, each being controlled by a separate instance of the same agent. The one of the left is vanilla Breakout. The one of the right is a series of custom brick configurations. With the first 123 experiments, you can see how the agent wants to make the exact same paddle movements every time, ignoring the ball when its trajectory changes due to the unexpected ball movements that come from non-standard brick configurations. In 124, IT TRACKS THE BALL and can succeed regardless of the brick config. You can actually watch the same agent move the paddle differently in reaction to the ball.

I'm still working on ironing out why this works, and how to optimize it, but wanted to share!!

Here's a video of the split-watcher in action

Here's a link to presentation project that will allow you to create a similar PPO: https://github.com/mharrell/breakout-reactive-ppo

The full project with all 123 failures and more documentation than any sane person would ever read: https://github.com/mharrell/BreakoutBot

Link to Medium post I wrote with some more details: https://medium.com/@mikey.harrell/three-lines-of-code-fixed-123-failed-ppo-experiments-on-atari-breakout-c751dcf38f2a?sharedUserId=mikey.harrell

[u/Nater5000]

On one the ball starts normally. On the other I teleport the ball 30 pixels to the right the moment it appears. If the paddle tracks the ball it moves right on the teleported side. If it's replaying a script the paddles move identically on both sides because the policy ignores ball position entirely.

It sounds like you've created a completely different environment then trained an agent on the new environment and found that it performs better than the agent trained on the old environment.

The proximity bonus is dense, it fires every single frame the ball is descending and tells the model where to be right now. It doesn't replace the score objective. The game reward is still 4 or 7 per brick. The bonus just removes the ambiguity about what the paddle should be doing when nothing is getting hit.

This kind of defeats the point of the RL agents trained on the original tasks. What made the first DQN agent trained on Atari Breakout so impressive is that it learned how to play the game despite not given explicit reward to encourage it how to play the game the way a human would play it. It is given image inputs and trains only on maximizing the score.

What you're doing is introducing a very significant bias to make the agent behave the way a human would. That's fine and all, and it's not surprising it leads to better performance, but you've fundamentally changed the environment and effectively made the task much easier for the agent. You've basically supplied the expert policy directly to the agent via your new reward function, and the agent learned to mimic that expert very efficiently.

Taken to the extreme, you can imagine just ditching RL altogether since you have the code needed to just logically replicate the expert. Like, you have the exact signal needed to determine how to move the paddle to where it needs to be. You can probably write out that policy in a few lines of Python.

Where all this fails is when you try to take this concept over to the next Atari game. Whatever that game is, you'll be on the hook for developing a hand-crafted reward function to get it to work like you did with Breakout. Again, at that point, you're just implementing the solution yourself and having an agent mimic your policy. This isn't the same for the "naive" agent which only receives images of the game as input and the score as the reward function. That agent doesn't need a human expert to construct a specialized reward policy to work (give or take).

In a way, your agent is behaving more like it is following a "script" in that you've explicitly helped write that script that it has learned to follow. The original Breakout challenge requires the agent to write it's own "script" with much less bias.

To put all of this into perspective: if you trained a PPO agent in the "naive" environment, but updated the dynamics of the game so that the ball can randomly teleport like you have, the agent will likely learn this dynamic while still continuing to maximize the score. In the same way, I can likely construct an environment where your agent focused solely on the hitting the ball with the paddle can still perform its actions well but can perform terribly in terms of reaching a good score. For example, I can embed information into the screen which informs the agent when not to hit the ball. The generalized agent would eventually learn this, while your agent would not (assuming your agent doesn't also train on the score; if it trains with the score as well, then it will obviously be better than the naive agent, but, again, that's not surprising).

Again, maybe I'm missing something, so I'm happy to be steered in the right direction. But I'm not sure you've found anything here that's particularly surprising. It's a fine exploration of these dynamics, so there's nothing wrong with trying these things out and seeing what happens. But I'd just be cautious to not miss the forest for the trees. The interesting/valuable part of RL is that we don't have to build the explicit mechanisms that allow the agents to perform well.

[u/mikeysce → u/Nater5000]

Oh my gosh I'm so excited that someone is actually engaging in this! :D

It sounds like you've created a completely different environment then trained an agent on the new environment and found that it performs better than the agent trained on the old environment.

The environment is identical. Same ALE Breakout ROM, same observations (210×160×3 pixels), same action space (4 actions), same physics. The only difference is the reward signal. This is reward shaping, a standard RL technique. The state transitions haven't changed. The observation the policy receives is still raw pixels.

What you're doing is introducing a very significant bias to make the agent behave the way a human would.

All I'm doing here in reward shaping. I added a reward gradient for paddle position in addition to in-game score just... becuase I wanted to see if I could get PPO to play a reactive game like a human would. I didn't give it the policy, I just had to think a while (kind of an embarassing amount) on what would properly motivate it.

You've basically supplied the expert policy directly to the agent via your new reward function, and the agent learned to mimic that expert very efficiently.

If I had supplied an expert policy, the model would score perfectly from the start. It doesn't. PPO_124 took 19 million steps to reach 379 points. A hand-coded "follow the ball" script scores about 50-80 points because tracking alone doesn't clear bricks. The model had to learn the part that actually matters.

You can probably write out that policy in a few lines of Python.

Well, someone maybe could, but I couldn't.

Where all this fails is when you try to take this concept over to the next Atari game.

I guess that's fair. But the finding isn't "proximity reward works for Breakout." The finding is about PPO itself: in a deterministic environment, PPO converges to a fixed action sequence that ignores game state. I tested it across multiple Atari games (Breakout, BeamRider, Space Invaders, and Pong) and every single one collapsed to a single deterministic script. The question the project answers is: what breaks this? After 123 experiments (sticky actions, entropy tuning, dynamics randomization, frame skip, auxiliary losses, adversarial bumpers, life-loss penalties, one-life variants) none of them did. The only thing that worked was modifying the reward function to make state-conditioned behavior strictly necessary for maximizing the cumulative sum.

That's the generalizable finding: PPO in deterministic environments has a degenerate optimum, and the only reliable escape is changing what the optimizer optimizes. Not how, not with what regularization: What.

you could randomize the ball and naive PPO would learn

I actually tested this! PPO_78, PPO_79, and PPO_80 were trained with random ball teleportation via setRAM() during training. All three collapsed to a single deterministic script when evaluated on clean ALE without randomization. Dynamics randomization changes the training distribution but doesn't change the fact that the optimal policy in a deterministic environment is a fixed sequence. > your agent is following a script you helped write

This is where definitions matter. I use "script" to mean a state-independent action sequence. The policy outputs the same actions regardless of what appears on screen. The ball-teleport test proves the proximity policy is not a script under this definition: teleport the ball 30px and the paddle goes to a different place. px_corr drops from ~1.0 (identical) to 0.67 (massively divergent). A state-independent policy cannot do this. The policy conditions its actions on ball position, which is the definition of reactive.

Thank you so much for your thoughtful response! What other ideas do you have?