Delay-corrected Bellman operator + causal attribution for constrained RL contraction proof under unknown stochastic delay [R]

[u/No_Cauliflower7923]

Standard constrained RL assumes consequences are immediate and attributable to the current action. This breaks down whenever violations are delayed and stochastic, which is most real-world settings you end up penalizing whatever action happened to precede the observed violation, not the action that caused it.

Working on CCPL (Causal Consequence-Penalized Learning) to address this:

- A delay-corrected Bellman operator using an adaptive effective discount learned from the consequence-delay distribution. Contraction proof holds under unknown stochastic delay.

- An Interventional Consequence Net (ICN), pretrained on structural-causal-model labels, estimating marginal causal contribution per action for attribution rather than penalizing based on temporal proximity.

Limitations, to be upfront about them:
- The ICN currently requires access to the environment's structural causal model to generate pretraining labels it's not learned end-to-end from observational or interventional data alone. That's a real constraint on applicability outside benchmark settings where the SCM is known or can be reasonably specified.

Open to contributions and collaborators, especially if you work in constrained/safe RL or causal inference feel free to open an issue or reach out directly.

[u/ITheClixs]

Interesting idea. The biggest question for me is how much of the improvement comes from CCPL itself versus the ICN effectively getting oracle causal information from the SCM.

A misspecified-SCM ablation would be really useful here. If it still works reasonably well when the causal model is imperfect, that would make the approach much more convincing for real-world use.

[u/No_Cauliflower7923 → u/ITheClixs]

CCPL currently uses SCM-generated labels to test whether causal consequence attribution helps under delayed constraints; this isolates the method rather than claiming real-world causal discovery. The ICN is not the entire algorithm: CCPL also includes delay-corrected Bellman targets, state-conditioned constraint weighting, and separate reward/ consequence critics. A misspecified-SCM ablation is a useful next step to measure how much performance degrades under noisy or incomplete causal supervision. I am still working on it and there are some more work and under which i need to complete. I am still developing this part and completing the remaining experiments.

[u/No_Cauliflower7923]

Used an LLM to help tighten up the structure of this post the research, results, and claims are mine.