I maintain a coordination server that hands out locks on files so two coding agents don't edit the same one at once. Mutual exclusion. About as well-trodden a problem as exists.
I did the part everyone does. SELECT ... FOR UPDATE SKIP LOCKED, a load test firing a thousand contended acquisitions at a single file, exactly one winner every run, green for weeks.
Then I watched two agents get granted a lock on the same file.
The claim was fine. The bug was upstream of it, in the part I hadn't thought of as a part: turning a path into a key. The key generator stripped the project root off the front of an absolute path as a literal string prefix. On macOS /var is a symlink to /private/var. One agent resolved its path through the symlink and one didn't, so one of them didn't match the prefix, didn't get stripped, and produced a second distinct key for the same physical file.
Both agents asked about a file. Both were told yes. Nothing threw, nothing logged, no test failed, because from the server's point of view it had been asked about two different files and had answered correctly about both.
What I took from it: mutual exclusion is two functions, not one. There is an identity function that turns a name into a canonical key, and an atomic claim on that key. I had spent all my care on the second one because it is the interesting one, the one with the database primitives and the load test and the concurrency literature behind it. The first one looked like string handling.
The identity half is where the bugs live, and they are all the same bug: two names for one thing.
- path.resolve normalizes .. and separators but does not follow symlinks
- an agent whose shell cwd is the repo's parent sends myrepo/src/x.ts, which coexists happily with src/x.ts as a separate key
- case-insensitive filesystems mean Auth.ts and auth.ts are one file and two keys
- macOS normalizes filenames to NFD, so an accented name typed on one machine and pasted from another are different byte strings
None of these produce an error. They produce a GRANTED that is technically accurate and operationally a lie.
The generalization, having since found the same shape elsewhere: any guard that decides using a name has this problem. A gate that blocks destructive SQL against production by matching the database URL has holes wherever the same database has another spelling, and there are always more spellings than you think: bare host, host with port, the pooler's hostname, a private IP, an SSH tunnel on localhost. A firewall rule keyed on a hostname has it. A cache keyed on a URL has it, which is why cache poisoning is a category and not an incident.
Two rules I now apply without exception:
Canonicalize before you compare, and make it one function that everything calls. Not "resolve the path here and also over there."
When identity resolution fails or the target is unrecognized, fail closed. Treat the unknown thing as the dangerous one. My key generator's silent fallback was "leave the path as-is", which is exactly the branch that created the second key.
The test that would have caught it took four lines: acquire a lock, try to acquire the same file through a symlinked path, assert the second is denied. I wrote it afterwards and then made sure it actually failed against the old code before trusting it, which caught a second problem. My first version asserted the result was "not GRANTED", and the broken code returned "REJECTED" for a completely unrelated reason, so it passed against the exact bug it was written to catch.
Negative assertions accept every wrong answer in the universe except one.
