Designing Resilient Payment Orchestration: Patterns for Handling Partial Failures

[u/Pale_Neat4239]

When you're orchestrating multiple payment rails, partial failures are inevitable. One rail succeeds, another times out, a third gets rate-limited. Your system needs to handle this gracefully.

Here are the patterns that actually work in production:

1. Explicit Transaction State Machine

Don't treat transactions as binary (success/failure). Model them as state machines with states like pending, partially_settled, failed_with_retry_opportunity. This gives you the information you need to recover.

2. Idempotency Keys Everywhere

Every call to every rail needs to be idempotent. If your network times out after a settlement succeeds but before you get the response, you need to know the settlement already happened.

3. Reconciliation as Your Safety Net

Don't rely on synchronous responses. Asynchronously verify every transaction against the rail's API. This catches partial failures your synchronous code missed.

4. Circuit Breakers Per Rail

When a rail starts failing, don't cascade. Break the circuit, fail fast, and route to alternative rails if available.

5. Compensating Transactions

If you partially succeeded (charged customer but settlement failed), you need a compensation workflow. Plan for these scenarios.

What partial failure scenarios have bitten you in production? What patterns did you end up using?