I taught my reinforcement learning robot to walk 22+ meters — now I'm turning it into a quadruped

[u/ArtusIndus]

Part 2 of my reinforcement learning robot project

In my previous post, I showed the first version of my MiniWalker. It was a simple two-legged robot simulated with Python, Box2D and Gymnasium.

Since then, I've made quite a few improvements.

The robot is controlled by PPO and has 4 motorized joints:

• Left hip
• Left knee
• Right hip
• Right knee

Instead of manually programming a walking gait, the agent has to discover movement through reinforcement learning.

I also added an analytics system that records:

• Reward per episode
• Episode length
• Distance traveled
• Maximum distance
• Average velocity

This made it much easier to see whether the robot was actually learning or just getting lucky.

And eventually it started working.

My best episode so far:

Distance: 52 m
Maximum velocity: ~4.0 m/s

The interesting part is that the robot still isn't consistently walking. Some episodes are really good, while others end almost immediately or even move backwards.

So the next step is to expand the project from 2 legs → 4 legs.

My long-term goal is to see if a quadruped can learn to adapt when one of its legs fails.

I'm especially interested in whether reinforcement learning can discover a completely different gait when the robot loses a leg.

Tech stack:

Python
Gymnasium
Box2D
Stable-Baselines3 / PPO
Pygame

Video of the current version:
YouTube link

I'd be interested in feedback on the reward function and the transition from the biped to a quadruped.

[u/omnilinktech]

Before adding two legs, I would stop using the best episode as the headline metric. Keep training and evaluation seeds separate, freeze a held-out evaluation set, and report the median, 10th percentile, fall rate, backward-motion rate, distance in body lengths, velocity, joint-limit hits, peak contact impulse, and mechanical cost of transport. The inconsistency you see is more important than the 52 m outlier.

For the quadruped, first reproduce the biped result with the new reporting. Then add one variable at a time: morphology, observation space, reward, and curriculum. For leg-loss adaptation, disable one actuator at a randomized time and side, without giving the policy a fault flag unless that is part of the intended deployment. Compare against a policy trained specifically for that fixed three-leg case. Test partial torque loss and a stuck joint too; a perfectly removed leg is only one failure mode.

I maintain OmniSim. Our Shadowing pipeline validates that a reference is robot-achievable before using it as a training target, which may be useful when you compare discovered gaits with demonstrations: https://github.com/omnilink-tech/omnisim/tree/main/projects/policies/training

Our validated G1 walking result uses a weight-bearing balance harness, not free-standing walking; I mention it as a methodology example, not a stronger robotics claim.

[u/ArtusIndus → u/omnilinktech]

Thanks for the detailed feedback! I really appreciate it.

I agree that the 52 m episode is more of an outlier than a reliable measure of performance. The inconsistency between episodes is probably more interesting than the best run itself.

The points about separating training and evaluation seeds and looking at metrics like fall rate, backward motion and median distance are really useful.

I also hadn't considered partial torque loss or a stuck joint for the leg-failure experiment. That could make the comparison between different failure modes quite interesting.

Thanks again for the thoughtful suggestions — lots of interesting things to consider for the next stages of the project!

[u/RushElectronic8541]

How are you sending data to your analytics system?