Replication

Holding data on a single server keeps management simple. There is no question about where anything lives and no worry about values disagreeing. At small scale that is the better arrangement, and plenty of services never outgrow it.

Two things catch up with that one server. The first is reads piling up. At Pagedeck, 80 percent of all requests were queries loading a document list. When everyone arrived on Monday morning the queries backed up, and because the same server was taking writes, anyone who had just pressed save waited alongside them.

The second is that server stopping. The day a disk failed, no document opened for two hours. Backups existed, but restoring took that long. The lifespan of the service was the lifespan of one machine.

Both problems point the same way: put the same data on another server too. With a copy in place, reads can be split off, and if the primary dies a copy can take its place. The price is that the copy has to be kept in the same state as the original, and doing that keeping is the whole content of this method.

It settles into place in six steps.

1. Assign roles. One server takes writes and several copies take reads. Relational databases generally support this arrangement out of the box. Having several servers accept writes requires a way to resolve two of them editing the same row, so most setups start by funnelling writes to one place and add copies from there.
2. Choose how changes travel. Either the write is not reported as complete until a copy confirms it, or the write lands on the primary, reports complete immediately, and reaches the copies afterward. The first loses nothing if the primary dies suddenly but slows every save by the copy's response time. The second is fast and leaves the copies anywhere from a few hundred milliseconds to a few seconds behind. Most teams start with the second and move specific tables to the first when losing a write would be unacceptable.
3. Copy everything once, at the start. Move the primary's current state onto the copy, then replay the change log from the moment the copy began so it catches up. With a large dataset this step alone runs for hours, and it has to happen without stopping writes on the primary, which is why databases ship a dedicated mechanism for it.
4. Split the reads. Keep separate read and write connections in the code. One distinction matters here: a screen showing something just saved reads from the primary, not a copy. Read from a copy right after a write and the document someone just created is missing from the list.
5. Watch the lag. Continuously measure how many seconds a copy trails the primary and get alerted when that crosses a threshold. While the gap is open, screens reading from copies are showing stale values. Lag is not constant either — it grows during bulk imports and shrinks again afterward.
6. Write down the procedure for the primary dying, in advance. Which copy gets promoted, whether that judgment is automatic or human, and how the remaining copies get pointed at the new primary. Without that document, the decision gets made for the first time on the night of an outage.

Three places fit: services where reads far outnumber writes, anything that has to keep serving when one machine stops, and running heavy analytical queries. The third comes up often — a query aggregating last quarter's usage runs for minutes, and running it on a copy keeps customer-facing screens fast. The same query on the primary would compete with every customer request.

Three places do not.

1. When writes are the bottleneck. Ten copies still leave one place accepting writes. Splitting writes means sharding, which breaks the data into pieces across servers.
2. When the data no longer fits on one machine. Replication holds the same thing several times over, so total capacity does not grow.
3. When most screens have to read what was just written. Nearly all reads end up routed to the primary and the copies sit idle.

There are three limits. First, the lag is visible to users. A document appearing in the list a few seconds late reads as a bug, so which screens read from copies has to be decided deliberately. Recording that decision per screen is cheaper than debugging it later. Second, the primary's load does not drop: it still takes every write, plus the work of shipping change logs to each copy. Past some number of copies, adding more increases lag. Third, the boundary of a transaction, a group of operations that must all succeed or all roll back, holds only on the primary. Copies receive the outcome afterward, so a read can land mid-sequence.

"Let's route queries to the copies. The primary backs up every Monday morning."

Move read load onto the replicas.

"The post-save screen has to read from the primary. Lag makes it look missing."

Values just written cannot be read from a copy.

"Replication isn't a backup. Deletions replicate too."

Correcting the roles of two different mechanisms.

At Pagedeck, Theo runs two copies and routes queries to them. He keeps a list of which screens read from the primary and which from a copy, and having new screens consult that list first stops the judgment from varying by developer. Lag sits on a dashboard and alerts past three seconds — not a law, just the point where the list screen starts looking wrong.

The promotion procedure gets rehearsed once a quarter. When it has never been run, an outage starts with someone checking whether the document is even correct.

Treating replication as a backup is the most common mistake. The same data sits on several machines, so it looks safe. But what replication moves is the change itself. Drop a table by accident and that deletion reaches the copies within seconds. Damage from human error or a bad deploy is not something replication prevents. What covers that is a backup you can roll back to a point in time, and neither substitutes for the other. A team with replication and no backup has redundancy against hardware and none against people.

The second is that adding copies makes everything faster. Reads genuinely split, but writes stay in one place, and each additional copy adds change-log traffic for the primary to ship. Past some point another copy buys a little read capacity and costs lag. If writes rather than reads are the wall, the answer is a different structure, not a higher copy count.