Concurrency

There was a time when a server handled one request at a time: finish the first one completely, then pick up the next. The trouble was how little of a request is actual computation. A request that loads a document list asks the data store to go find something and then waits 40 milliseconds for an answer. For those 40 milliseconds the server does nothing at all, and everyone queued behind it waits too.

Waiting dominates the work. Of the 50 milliseconds a request spends inside a server, maybe 5 are computation and the other 45 are spent waiting on a database or another service to reply. With ten people arriving at once, the tenth waits through all nine of those idle stretches before getting a turn.

Pagedeck ran into this directly. After lunch, when everyone opened documents at the same time, the list screen froze for several seconds. Theo looked at the server and found it wasn't short on processing power at all. The machine was idle most of the time — it simply wasn't picking up the next request while it sat idle.

Concurrency doesn't run several pieces of work in the same instant; it slips other work into the gaps where one piece has stopped to wait. Running two computations in the same instant requires two processing units, and that goes by a different name — parallelism. Concurrency holds up with a single unit.

It works the way one server covering five tables works. The order from table one goes to the kitchen, and instead of standing there until the food appears, the server takes an order at table two and refills water at table three. Five dishes aren't being cooked by the server at once; the waiting is being shared.

There are three common ways to arrange the interleaving.

1. Give each piece of work its own execution thread and let the operating system switch between them. It's the familiar approach, but each thread carries memory with it, so thousands of them get expensive.
2. Run a single thread that walks a list of pending operations and picks up whichever one has finished. Rather than waiting for a reply, it notes the pending call, moves on, and resumes when the answer lands. This handles the most requests on the least hardware when waiting dominates.
3. Run several copies of the program and split requests between them. Nothing is shared in memory, which is safer, but each copy costs a full set of resources.

All three carry the same price: order stops being guaranteed. Processed one at a time, B starts after A finishes. Interleaved, B can read a value in the window after A read it and before A wrote it back. That window is where a race condition comes from. So interleaved designs come with countermeasures — idempotency, the property of a request producing the same result no matter how many times it arrives, or a lock that lets only one piece of work through a section at a time.

The first place you meet it is request handling. A server — the program that takes requests and returns responses — almost always interleaves them already. Nobody turns it on; it's the default, so reading the code tells you nothing about it. Most developers first notice the interleaving on the day it goes wrong.

The second is work split off from the response. If exporting a 300-page document takes 40 seconds, nobody can hold a user there for 40 seconds, so the server acknowledges the request immediately and does the work behind it. Running several of these background jobs at once needs concurrency, and a message queue is what lines them up when a crowd of them arrives together.

The third is calling outward more than once. If billing details, usage, and team information each come from a different service at 300 milliseconds a call, doing them in sequence costs 900 milliseconds and interleaving them costs 300. None of the three has any reason to wait for the others.

It's worth knowing where it buys nothing. Work that computes without waiting doesn't get faster when interleaved. Ten heavy calculations rotating through one processing unit take the same total time, and each one finishes ten times later. That problem is solved by adding units.

"At 3,000 concurrent connections the queue backs up. We need more workers."

A report that the amount of interleaved work has hit its ceiling.

"This is a concurrency bug, so I can't reproduce it. It shows up maybe one run in twenty."

The failure needs a specific overlap in timing to appear at all.

"In sequence it's 900 milliseconds. Interleaved it finishes in 300."

A proposal to overlap calls that have no reason to wait on each other.

At Pagedeck the word came up most sharply over workshop signups. A workshop had one seat left, two people pressed the signup button in the same instant, and because the two requests were interleaved both read the remaining seats as 1. Both signups were accepted and the seat count went to minus one. When Sofia passed the complaint along, Theo pressed the button ten times trying to recreate it and never saw it happen once.

Once the cause was clear, the fix wasn't to stop interleaving requests. It was to let only one request at a time through the section that reads and writes the seat count. Removing the interleaving would have taken the server's capacity down with it.

The most common reading is that turning on concurrency makes things faster. Handling many things at once certainly sounds like speed. But the gain only exists where waiting exists. Calling a data store and waiting for the answer leaves a gap another task can fill, and total time drops; pure computation interleaves into exactly the same total. Restructure code without that distinction and you get more complexity at the same response time.

The other is assuming the server handles all of it, so there's nothing to think about. That holds up for a while — most server programs interleave by default and months can pass without incident. But the moment two requests touch the same value, order starts changing the outcome. Left alone, this comes back once or twice a month as data that can't be explained. Inventory goes negative or a charge goes out twice, and every individual request looks correct in the logs.