Race Condition

As long as requests arrive one at a time, most code behaves correctly. The trouble starts when two requests touch the same data in the same instant, and at that point reading the code reveals nothing wrong with it.

Pagedeck hit this. The free plan allows three documents, and a free account with four documents turned up in production data. Theo read the code and the check was plainly there. The cause was a user pressing the new document button twice in quick succession. Both requests arrived almost simultaneously, both were told the account currently had two documents, and both concluded that a third was allowed.

What was wrong was neither the check nor the write, but the fact that time passed between them. This class of fault gets its own name because the cause can't be pointed at as a line of code. The cause is two requests arriving as a dead heat.

The fault lives in the gaps between reading, deciding and writing. If the value one request read changes before it writes, that decision was based on a figure that has already expired.

Laid out in order, the document limit check goes like this.

1. Request A reads the document count. It's 2.
2. Request B reads the document count. Still 2.
3. A decides 2 is under the limit and writes. The count becomes 3.
4. B also decided against 2, and writes. The count becomes 4.

Arrive slightly later and B would have re-read between steps three and four and been refused. That the outcome depends on the gap between arrivals is the defining property here.

Ordering isn't guaranteed for several reasons at once: there are multiple servers, so two requests get handled by different ones; each server interleaves many operations; and the round trip to the database takes a different amount of time per request. None of those are things application code can observe.

Preventing it means removing the gap, or making the outcome identical whether or not the gap exists.

1. Put the rule in the database. A constraint that refuses a duplicate value means one of the two requests fails at write time, even though both passed the check in the code.
2. Bind the read and the write together. Wrap them in a transaction — a set of operations that either all succeed or all roll back — and lock the row so nothing else touches it meanwhile.
3. Don't read a value, compute, and write it back. Instruct the database to subtract 1,000 rather than writing the result of balance minus 1,000, and there's no interval to slip into.
4. Make a repeated request count once. Attach a unique token to each request and ignore tokens already processed; that property is idempotency.
5. Force an order. Put work in a message queue so operations on one account are processed strictly one at a time.

Disabling the button on screen is not on this list. It reduces double clicks and leaves two open tabs, refreshes, and automatic retries on a slow connection untouched. Rules hold only when they live on the server.

The term gets used more often to suspect a cause than to explain one. When data sits in a state the rules forbid and the code looks fine, this is the first name anyone reaches for. Four places produce most cases.

1. Limit checks. Three documents on a free plan, seats on a paid plan — anywhere a count gates an action. There is always an interval between counting and creating, and the tighter the limit the more often somebody sits right at its edge.
2. Inventory and first-come offers. This is how a coupon capped at 100 gets issued 103 times, and the three extra people have a valid code you now have to honour or revoke.
3. Payments. The same charge going through twice, where the loss is money and the refund is a support conversation, which is why idempotency usually lands here first.
4. Background work colliding with user requests. A job aggregating usage at 3am while someone deletes a document produces an aggregate that doesn't add up, and nobody notices until an invoice is questioned.

Detection usually happens in the data. Run a periodic check for rows that violate the rules — free accounts holding four or more documents, for instance. Since reproduction is hard, logging request arrival times down to the millisecond earns its keep in exactly this situation.

"I can't reproduce it but the data is there. Looks like a race condition."

No cause found in the code, and concurrent requests are the suspect.

"There's a check and no unique constraint, so it leaks."

The rule exists in the code and not in the database.

"Disabling the button is a workaround. It has to be blocked on the server."

Fixing it on screen leaves the same fault reachable by other paths.

Theo's fix for the document limit had two parts. Instead of counting documents per account, a unique constraint on account and document number made the fourth write fail in the database, and each request began carrying a token so a repeat of the same token was ignored. Then he swept six months of data and cleaned up the eleven accounts that had broken the rule. Two days went to the fix and three to the cleanup.

The most common belief is that low traffic means immunity. With ten concurrent users, two requests landing in the same instant looks unlikely. But this fault needs two requests, not two people. One person double-clicking, an app retrying automatically on a slow connection, or two open tabs is enough. On a small service the incidents are rarer and the cause is harder to find, because there is no traffic pattern to point at.

The other habit is reading "can't reproduce" as "fixed." Repeat the action twenty times, see nothing, and it feels resolved. In reality the interval is milliseconds wide, which is nearly impossible to hit by hand, and in a development environment requests are often processed one at a time so the interval never opens at all. Confirm this class of fault by checking that the rule is enforced in the database and that sending the same request twice produces one result. With both in place, it's blocked whether or not it reproduces, and that is the only form of confirmation worth trusting here.