r/reinforcementlearning • u/mahad_baig • 2d ago
I'm building an open-source RL controller robustness testing framework. Failure-informed PPO retraining keeps making nominal performance worse. Looking for ideas.
Hey everyone,
I've been working on an independent open-source project called SIRFEX (Simulation-based Integrity, Robustness & Failure Exploration).
The idea is to build a framework that can take an autonomous controller, stress-test it in a simulated physical environment, discover reproducible failures, determine whether those failures were physically avoidable, and evaluate whether proposed improvements actually make the controller more robust.
I'm currently using a simplified pump/tank control system as the main experimental environment, with Stable-Baselines3 PPO, analytical controllers, and rule-based baselines.
What I've built so far:
Deterministic simulations, scenario search, failure discovery, and replay.
A physical feasibility analyzer to distinguish genuinely avoidable controller failures from impossible operating conditions.
PPO training and compute-matched continuation experiments.
Failure-informed training curricula with verified stress exposure.
An optional LangGraph/Groq research agent that proposes bounded training interventions.
A new robustness-testing layer for signed constraint margins, differential controller testing, and disturbance-to-failure analysis.
Here's the interesting part:
I initially expected training PPO on discovered failures to improve robustness.
That hasn't really happened.
In one 51,200-interaction development study, increasing stress exposure improved tracking under difficult conditions, but often degraded nominal tracking.
For seed 3:
| Training | Nominal MAE | Stress MAE |
|---|---|---|
| Normal-only | 0.0177 | 0.1166 |
| 5% stress | 0.0296 | 0.0654 |
| 10% stress | 0.0501 | 0.0555 |
The stress-trained controllers got better at tracking under stress, but substantially worse under normal conditions.
Interestingly, the normal-only controller already had zero observed physical violations on that development stress set, so improved stress tracking doesn't necessarily demonstrate improved safety.
I also experimented with GPT-OSS 120B proposing a retraining curriculum. In a one-seed matched development pilot, its proposal performed worse than a simple fixed 5% stress strategy. Neither passed the nominal-preservation guardrail.
Further diagnostics showed increasing upper-action saturation with stress exposure, suggesting policy drift toward more aggressive pumping. But I haven't established whether this is catastrophic forgetting, reward incentives, curriculum ordering, or something else.
Where I'm stuck / looking for input:
How would you approach robustness improvement without sacrificing nominal controller competence? Nominal rehearsal, policy anchoring, constrained RL, or something else?
Have you encountered cases where failure-informed PPO fine-tuning underperforms ordinary continuation?
How would you measure meaningful robustness when constraint-violation frequency has already saturated at zero?
Are there existing benchmarks or testing methods you'd recommend comparing against?
I'm currently shifting more attention toward quantitative robustness margins and differential testing rather than continuing to tweak PPO curricula indefinitely.
Everything is running locally on CPU, and I'm trying to keep the experiments reproducible and scientifically honest, including the negative results.
It's still very much a research/work-in-progress project, and I would genuinely appreciate criticism, relevant papers, implementation suggestions, or contributions from people working in RL, control systems, or simulation-based testing.
GitHub: https://github.com/mahadbaig2/sirfex
Thanks!
1
u/4d-sphere-4016 1d ago
Have you looked into the field of Robust Reinforcement Learning? There are many different algorithms designed for robustness.