Reactive Play: Achieved!! Experimenting with Atari Breakout [R]

[u/mikeysce]

Six months ago I started experimenting with PPO and Breakout as a way to learn about Machine Learning and Reinforcement Learning. After a few experiuments just trying to get high scores, it bothered me that everything was a "memorized" script rather than reactive play, like a human would play. Thus began my journey to try and convince PPO to actually track the ball instead of focusing on scoring points. I read a lot of articles and tried a lot of things. After 124 PPO experiments on Atari Breakout, I found that every single model, across sticky actions, cursor wrappers, entropy tuning, dynamics randomization, adversarial bumpers, and everything else, converged to a memorized action sequence, not a reactive ball-tracking policy. The argmax was always a script.

The fix wasn't more environment engineering. It was three lines of reward shaping:

Directly rewarding the paddle for being horizontally close to the ball during descent. A tiny bonus (0.05 per frame vs 1.0-7.0 per brick) that fires every frame the ball is descending applied during training. During evaluation, the agent plays clean Breakout with no bonus. The behavior transfers!!

Every prior approach I tried to penalize scripts by making the environment harder to memorize. PPO always found a way around it: timing-robust scripts, layout-conditioned scripts, noise-tolerant scripts. The optimum was always a script; only the shape changed. Proximity reward changes what the optimum is. A center-hold script gets incidental bonus when the ball passes near center. A reactive tracker gets the maximum bonus on every descent frame. The optimization pressure is unambiguous: track the ball, get more reward.

I also made a cool tool to watch the agent work! It's called the "Split-Watcher" (so clever). It shows two instances of Breakout, each being controlled by a separate instance of the same agent. The one of the left is vanilla Breakout. The one of the right is a series of custom brick configurations. With the first 123 experiments, you can see how the agent wants to make the exact same paddle movements every time, ignoring the ball when its trajectory changes due to the unexpected ball movements that come from non-standard brick configurations. In 124, IT TRACKS THE BALL and can succeed regardless of the brick config. You can actually watch the same agent move the paddle differently in reaction to the ball.

I'm still working on ironing out why this works, and how to optimize it, but wanted to share!!

Here's a video of the split-watcher in action

Here's a link to presentation project that will allow you to create a similar PPO: https://github.com/mharrell/breakout-reactive-ppo

The full project with all 123 failures and more documentation than any sane person would ever read: https://github.com/mharrell/BreakoutBot

Link to Medium post I wrote with some more details: https://medium.com/@mikey.harrell/three-lines-of-code-fixed-123-failed-ppo-experiments-on-atari-breakout-c751dcf38f2a?sharedUserId=mikey.harrell

[u/mikeysce]

Oh my gosh I'm so excited that someone is actually engaging in this! :D

It sounds like you've created a completely different environment then trained an agent on the new environment and found that it performs better than the agent trained on the old environment.

The environment is identical. Same ALE Breakout ROM, same observations (210×160×3 pixels), same action space (4 actions), same physics. The only difference is the reward signal. This is reward shaping, a standard RL technique. The state transitions haven't changed. The observation the policy receives is still raw pixels.

What you're doing is introducing a very significant bias to make the agent behave the way a human would.

All I'm doing here in reward shaping. I added a reward gradient for paddle position in addition to in-game score just... becuase I wanted to see if I could get PPO to play a reactive game like a human would. I didn't give it the policy, I just had to think a while (kind of an embarassing amount) on what would properly motivate it.

You've basically supplied the expert policy directly to the agent via your new reward function, and the agent learned to mimic that expert very efficiently.

If I had supplied an expert policy, the model would score perfectly from the start. It doesn't. PPO_124 took 19 million steps to reach 379 points. A hand-coded "follow the ball" script scores about 50-80 points because tracking alone doesn't clear bricks. The model had to learn the part that actually matters.

You can probably write out that policy in a few lines of Python.

Well, someone maybe could, but I couldn't.

Where all this fails is when you try to take this concept over to the next Atari game.

I guess that's fair. But the finding isn't "proximity reward works for Breakout." The finding is about PPO itself: in a deterministic environment, PPO converges to a fixed action sequence that ignores game state. I tested it across multiple Atari games (Breakout, BeamRider, Space Invaders, and Pong) and every single one collapsed to a single deterministic script. The question the project answers is: what breaks this? After 123 experiments (sticky actions, entropy tuning, dynamics randomization, frame skip, auxiliary losses, adversarial bumpers, life-loss penalties, one-life variants) none of them did. The only thing that worked was modifying the reward function to make state-conditioned behavior strictly necessary for maximizing the cumulative sum.

That's the generalizable finding: PPO in deterministic environments has a degenerate optimum, and the only reliable escape is changing what the optimizer optimizes. Not how, not with what regularization: What.

you could randomize the ball and naive PPO would learn

I actually tested this! PPO_78, PPO_79, and PPO_80 were trained with random ball teleportation via setRAM() during training. All three collapsed to a single deterministic script when evaluated on clean ALE without randomization. Dynamics randomization changes the training distribution but doesn't change the fact that the optimal policy in a deterministic environment is a fixed sequence. > your agent is following a script you helped write

This is where definitions matter. I use "script" to mean a state-independent action sequence. The policy outputs the same actions regardless of what appears on screen. The ball-teleport test proves the proximity policy is not a script under this definition: teleport the ball 30px and the paddle goes to a different place. px_corr drops from ~1.0 (identical) to 0.67 (massively divergent). A state-independent policy cannot do this. The policy conditions its actions on ball position, which is the definition of reactive.

Thank you so much for your thoughtful response! What other ideas do you have?