Load Balancer

One server is enough while a service is small. The name in the address bar points at that machine, every request goes there, and that machine answers all of them. Trouble starts not when traffic grows but when it spikes. There is a ceiling on how many requests one machine can handle at once, and past that ceiling responses slow down and then stop coming at all.

The worse problem is that the one machine is the whole service. Reboot it and nobody can get in; the few minutes spent deploying a new version are the same; a failed disk means no service until it's replaced. A spot like that is a single point of failure — one thing whose failure stops everything.

Pagedeck hit this directly. On the day the free-plan announcement spread, concurrent traffic jumped to ten times normal and the server stopped responding. Theo had an identically configured second machine running within ten minutes, and it made no difference. Nobody knew the new machine's address, so everyone kept arriving at the dead one. Adding a server and getting traffic onto the server turned out to be two different jobs.

A load balancer connects the two. The address the outside world sees stays fixed, and however many machines stand behind it, incoming requests get spread across them. From then on, adding a server is a line of configuration.

Putting one place in front that receives requests and passes them along solves three separate problems at once.

1. Distribution — a rule decides which server gets the next request. The simplest rule takes them in turn. Where response times vary a lot, the request goes to whichever server currently holds the fewest open connections.
2. Fault isolation — the balancer asks each server behind it whether it's alive every few seconds and drops any machine that doesn't answer. A common setting checks every five seconds and removes a server after two consecutive failures, so within about ten seconds of a machine dying, nothing is routed to it. Drop one of four servers and the remaining three each absorb 33 percent more, which is why the fleet size gets picked with enough headroom to survive losing one.
3. Replacement without downtime — this is what makes a rolling deployment possible. Pull server 1 from the list, upgrade it, put it back, and repeat through servers 2, 3 and 4. The other three are serving traffic the whole time, so there is no moment of downtime from the user's side.

The catch is sessions. A session is one continuous stretch of use from arrival to exit, and if the signed-in state lives only in server 1's memory, the next request landing on server 2 is treated as a stranger. There are two fixes: pin each user to the same server for the duration, or move the signed-in state out to shared storage every server can read. Pinning takes less work, but everyone pinned to a server loses their session when it dies, and adding machines doesn't rebalance the users already pinned.

Three other approaches address this, and none of them fully replaces the device.

1. Make one server bigger — raising memory from 8 GB to 32 GB, say. It needs no code changes, which makes it the fastest move in an emergency. But one machine can only grow so far, it goes down while the resize happens, and the single point of failure is still there afterward.
2. Spread traffic with DNS — attach several server addresses to one name and hand out different ones to different callers. No hardware, but removing a dead server is slow. Whoever received an address holds on to it for a while, so traffic keeps arriving at the dead machine until that cached copy expires.
3. Put a reverse proxy in front — a relay that receives requests on behalf of the servers behind it, and one that usually ships with distribution built in. That overlap is why the same box often does both jobs in practice, and it's covered further down.

There is also the option of turning on what a cloud provider sells instead of running your own. It's billed hourly plus traffic processed, and the provider owns the problem of the balancer itself failing. Pairing this with autoscaling, where the machine count rises and falls on its own, effectively means using the managed version.

"I added two more behind the load balancer. We're at six now."

Capacity went up 50 percent without taking the service down.

"Server 3 failed its health check and got pulled. Let's find out why."

One machine stopped answering and was removed from the list; the service is currently running on the rest.

"People keep getting logged out — are we holding sessions in server memory?"

Suspecting a setup that loses signed-in state every time a request lands on a different machine.

After that day, Pagedeck put a load balancer in front and stood four servers behind it. Theo upgrades one machine at a time, and the fleet grows to six on its own over the lunch-hour peak and shrinks back to two overnight.

The common expectation is that once a load balancer is in place, adding servers keeps making things faster. Machine count and capacity look proportional, so the assumption is reasonable. But only the request-receiving side grows, and if every one of those servers is calling the same single database, the bottleneck simply moves there. This is where a fleet goes from 4 machines to 12 and the response time doesn't change. Add more machines in that state and the database collects more connections, which makes the whole thing slower. Horizontal scaling tracks machine count only when the work is built so any server produces the same result.

The other one is assuming the balancer itself doesn't fail. It was installed to prevent outages, so it feels like the sturdy part — but every request passes through that one point, and when it stops, a fleet of perfectly healthy servers serves nobody. Removing one single point of failure created a more certain one. That's why production setups run two or more of these, or use a managed product where the provider owns the redundancy.