Single Point of Failure

Teams spent years feeling safe once servers were plural. One web server becomes five, so one dying leaves four to absorb the traffic and nothing stops. Put a load balancer, which distributes requests across servers, in front and the picture looks finished. Most architecture diagrams stop being drawn at about that point.

Except all five are reading from one database. During the three minutes that database restarts at 4am, all five do nothing at all. The sentence written into the postmortem is nearly always the same: the servers were redundant and the database was one box.

The same thing happens away from servers. One person knowing the payment provider's password, deployment working only from one laptop, a domain registrar account tied to a departed employee's mailbox — all of these share the property. One thing disappears and everything stops. Single point of failure is the name that lets you go looking for that property on purpose.

A single point of failure shows up as a cell on the request path with no substitute. Finding them means laying the path out in a line and asking, at each cell, what happens if this is gone right now.

For a collaborative document product the path runs roughly like this: name resolution, the load balancer, the web servers, the database, the file store, and outside windows like the payment provider and the email sender. Three things get checked per cell.

1. Whether a partner exists — is there something to take over when this cell is gone.
2. Whether the handover is automatic — a partner that requires a human to press something is not a partner at 4am.
3. Whether it holds after taking over — when one of two dies, the survivor takes double. At 60 percent normal utilization that becomes 120 percent, and it dies shortly after, which is how a single failure turns into a full outage several minutes later.

Things that look redundant but are one copy are the most common finding. Three shapes recur: two boxes on the same power supply or in the same building; two boxes on the same account and payment method, so a suspended account takes both; and two boxes reading the same config file, so a bad file breaks them identically at the same moment. The third shape is the hardest to spot, because both boxes are genuinely separate until the day they aren't.

People and knowledge count as cells too. When one person is the only one who can perform a task, that person's holiday becomes incident response time. This cell doesn't shrink by adding servers — only by writing things down and distributing access, which is slower work and rarely appears on a roadmap.

Outside windows are cells you cannot make redundant. Tied to one payment provider, you stop when they stop. These get isolated rather than eliminated: document editing has to keep working while payments don't. Whether that isolation exists is usually decided long before the provider's first outage, in how the code was arranged.

The most common setting is a design review for new structure. Four uses come up.

1. Design review — draw the path and ask about a partner at each cell. Splitting a monolith, which packs everything into one deployable, into several pieces does not remove failure points on its own. When all five pieces read the same database, that database is still exactly one point of failure.
2. Postmortems — after an incident, ask which single thing stopped everything. The cell that answer names goes onto the next quarter's list, which is the one moment in the year when redundancy work gets prioritized without argument.
3. Setting commitments — putting uptime, the share of time a service ran normally, into a contract requires the failure points to be handled first. Promise 99.99 percent on a structure where one cell stops for three hours a year and the promise will not hold.
4. Preparing the business side — cells engineering cannot remove get handed to business continuity planning, which prepares for the business to carry on through an incident. Situations taking hours to recover, like an entire data center going down, get their procedure written in advance, because nobody drafts one calmly while it is happening.

Not every cell found gets fixed. Weigh the cost of a second copy against the loss from an outage. A cell that triggers contractual penalties after an hour gets fixed now; a cell where a day's outage ends in an apology email stays on the list. The list itself is the deliverable — it survives the people who wrote it.

"Five web servers and one database. That's the single point of failure."

Redundancy covers only part of the path.

"There's a partner but the failover is manual. Who presses it at 4am?"

Without automatic handover, a partner is the same as no partner.

"This deploy only runs from Theo's laptop. That's a failure point too."

Naming a cell that sits in a person and their knowledge rather than in a server.

At Pagedeck, Theo keeps the request path drawn as seven cells with each cell's partner and handover method noted. Last quarter the database moved to a structure with a replica, and wiring up automatic failover took three weeks. The email sender has no partner; instead it was isolated so that a sending outage delays notification email while document editing keeps working. Grace consults this table whenever an uptime clause enters an enterprise contract.

The most common mistake is reading server count as the cure. More boxes does mean one can die, so the reasoning is sound as far as it goes. But whatever those boxes share stays shared — the same database, the same config store, the same account, the same building's power. That's why companies with five web servers still go entirely dark twice a year, and why the postmortem reads as a surprise every time. The check runs on the path, not the count. Laying out the cells a request passes through and counting the ones without partners produces an answer far faster.

Believing every failure point must be removed is the other one. It carries the word risk, so removal looks mandatory. But the cost of a second copy differs per cell, and some cells — outside windows — aren't yours to touch. Try to remove them all and the year's budget goes while the list stays the same length. In practice it splits two ways: build partners for the expensive cells, and isolate the rest so the outage doesn't spread. Shrinking the blast radius is usually cheaper than eliminating the outage.