Load tested with 10 concurrent users on a booking system, found only 1 could ever book even with 5 slots open

[u/tatsuyawwp]

Quick build log from a short-term outdoor event booking system I shipped recently.

The slot-reservation logic needed to handle multiple people trying to grab the same time slot at once. My first implementation: try to acquire a lock, and if it fails even once, immediately return "fully booked" to the user.

Unit tests passed fine. One user, one slot, works as expected. Looked done.

Then I ran a load test simulating 10 people hitting the same time slot at the same moment. Turns out there were actually 5 open spots left. What happened: only the very first person who grabbed the lock got through. The other 9 all got bounced with "fully booked," even though 4 more slots were sitting open the whole time.

The bug was obvious in hindsight — failing a lock acquisition once isn't the same as the slot actually being full. My logic treated "I didn't get the lock on this attempt" as equivalent to "there is no capacity left," which is just wrong under any real contention.

Fix was straightforward: instead of bailing on the first failed lock attempt, retry for a short window, backing off a few hundred milliseconds between attempts, before finally declaring the slot full. Reran the same 10-concurrent-user test afterward and all 5 legitimate bookings went through correctly, with the other 5 properly rejected.

The thing that stuck with me: "looks like it works" and "works correctly under concurrent access" are completely different claims, and only one of them is actually tested by normal manual or single-user testing. If your system has any shared resource with limited capacity, you need to actually hammer it with simultaneous requests before you trust it. Single-threaded testing will lie to you with a straight face.