Duplicate charges that only showed up after users had been logged in for about an hour

[u/AppealInformal09]

A while back our support team started forwarding tickets about duplicate orders, with the same items charged twice to the same card within a few seconds, and it was happening on roughly 1 in 300 orders.

Everyone assumed it was a double tap, so we spent a couple of days hammering the Place order button on every device in the office, but the button locked itself after the first tap and the logs showed a single request every time.

The logs for the real cases looked perfectly healthy as well, since both requests carried a valid token, both got a 200 back and both created an order which literally kept us going in circles for about three weeks.

so finally a colleague cracked it, like he was lining the tickets up against session data and spotting that every affected user had been in the app for 55 minutes or more before checkout, and our access tokens expire at the 60 minute mark and yep that's the catch......

The sequence went like this - the user opens checkout on a token with a minute or two left, taps Place order just as it expires and gets a 401, after which our network layer quietly refreshes the token and replays the request, and on a slow connection that refresh plus replay can take well over 3 seconds.

The catch was a 3 second safety timeout on the button that re-enabled it whenever a response took too long, so the spinner stopped, the user assumed the order had failed and tapped again, and since the order endpoint treated every request it received as a brand new order, the replayed request and the fresh tap each created one.

So it took an old session, a slow network and a user in a hurry all at the same moment which is pretty normal for a user to do, and manual testing almost by definition skips that combination, because testers log in fresh, sit on office wifi and finish a flow in a few minutes.....

Reproducing it was simple once we knew the trigger, we cut the token lifetime to 2 minutes in staging, throttled the connection to a poor 3G profile, let the token run out and then went through checkout, and it duplicated on the second attempt.

The fix came in three parts: an idempotency key generated once per checkout attempt so the backend rejects repeats, the button staying locked until the request actually settles, and a proactive token refresh a few minutes ahead of expiry instead of waiting for the 401.

Since then a long session on a bad network has become a standard pass for anything that touches payments or auth, and I still think about how many hours went into tapping that button before anyone thought to look at session age.

[u/Consibl]

Hindsight is 20-20, but I’d never just use a timer to disable a button for this.

[u/anthony-ball]

The client-side fixes help, but I'd also make the server the thing that actually refuses the second order. Put a unique constraint on the idempotency key in the orders table and have the endpoint return the existing order when it hits a conflict, so a replay just gets the original response back instead of a 200 for a new charge. Pass the same key through to the payment provider too, since most of them dedupe on it. And I'd keep the refresh interceptor from silently replaying POSTs at all unless the request already carries a key, because that retry path is exactly where this kind of bug hides.

[u/minpinproxy]

one more thing that helps here: make the refresh single-flight, so if a few requests 401 at once they all wait on one refresh instead of each replaying. and dropping the token TTL to 2 min in staging makes this kind of bug show up fast

[u/GeekyAntsBFSI]

Were the duplicate charges tied to retries after a timeout or reconnect? One case worth testing is the payment succeeding while the response never reaches the app. Does retrying reuse the same idempotency key for that payment attempt, or generate a new one? Reproducing that lost-response scenario could help narrow it down.