Horizontal Scaling

For a long time there was only one answer to a service getting slow: swap the machine for a better one. Move the memory from 8 GB to 32 GB, put in a faster processor, and the single machine absorbed more requests. Nothing in the program had to change, which made it the easiest move available.

That move runs into two walls. The first is a ceiling. There is a largest machine anyone sells, and once you are on it, more money buys nothing — there is simply no bigger box to order. The second wall is what happens when that one machine stops. An expensive server whose disk fails takes the whole service down with it, and paying ten times as much does not make failure ten times rarer.

Pagedeck hit both walls on the same day. A signup promotion brought in six times the usual traffic, the document editor froze for fifteen seconds at a time, and when Theo went to bump the server up a size he found the team was already on the largest instance the cloud provider offered. There was nothing left to buy.

Horizontal scaling is the turn taken at that dead end. Instead of growing one machine, you run several machines of the same size and let them share the requests. The count can keep going up, so the ceiling disappears, and when one machine dies the rest keep answering, so a single failure no longer takes the service with it.

Start by making the program produce the same answer no matter which machine handles a request, then run several copies and put something in front of them to divide the traffic. Four steps.

1. Stop servers from holding per-user information inside themselves. If the fact that someone is logged in lives only in machine 1's memory, that person looks logged out the moment their next request lands on machine 3. A design that keeps no information between requests on the server itself is called stateless, and the per-user information moves to shared storage outside the machines.
2. Make every machine run exactly the same program. Installed by hand, machine 1 and machine 5 drift to different versions, and then a bug appears for some users and not others. Packaging the program with everything it needs to run, so it behaves identically anywhere, is what a container does, and stamping out copies from one removes the drift.
3. Put a load balancer in front of the machines. It takes each incoming request and hands it to one of the servers behind it. Because it pings each one on a schedule and drops any that stop answering, a machine can die without users noticing.
4. Decide how many machines to keep running. You can fix the number, or hand the decision to autoscaling, which adds and removes machines based on load. The second option drops to two machines overnight and climbs to eight at lunchtime, which also trims what you pay for idle capacity.

Even after all four steps, storage is still a single place. Ten machines all hitting the same database instance just move the bottleneck to that instance. When reads dominate, replication copies the same data onto several servers so reads can spread out; when the data itself outgrows one machine, sharding splits it into pieces that live on different servers.

It fits work where requests are independent and any machine produces the same result. Rendering a web page, resizing an image, sending a notification. A million requests split ten ways at a hundred thousand each come out the same as a million handled in one place. What it does not fix is a single long calculation that has to run in order — spreading that across ten machines does not make it finish in a tenth of the time.

There are three limits worth knowing before you commit.

1. Capacity does not rise as fast as the machine count. Going from two machines to four looks like double the throughput, but every request still waits its turn at shared storage, so the real gain lands closer to 1.7x.
2. Operational work grows with the fleet. A team used to watching one server now digs through logs from twenty and has to find which machine the error came from first. Without something that collects logs in one place, outages get longer rather than shorter.
3. Getting ready takes time. Pulling per-user information out of the servers is a code change, and it can run for weeks. If the traffic arrives tonight, buying a bigger machine is the realistic move.

That is why teams usually run both. Size up to put out the immediate fire, and use the time that buys to move the program toward a shape where machine count is the lever.

"We'll ride out the promotion with horizontal scaling — let's go up to six machines beforehand."

Add servers ahead of the event rather than reacting during it.

"This feature can't scale horizontally yet. It keeps the session in server memory."

Adding machines would log people out, so there is a code change to make first.

"We doubled the machine count and response time didn't halve. I think we need to look at the database."

The bottleneck is storage, not the application servers.

After Pagedeck signed its first enterprise customer, Theo spent two months walking exactly this checklist. Login information moved out of server memory into shared storage, the service went to three machines, and a load balancer went in front of them. Response time at 10 a.m. on a weekday dropped from 2.4 seconds to 0.9.

Speed was not the whole payoff. A month later a disk failed on one of the three machines. The old setup would have meant forty minutes of downtime; this time nothing reached users, and Sofia never had to write the incident note to the customer.

The most common mistake is expecting capacity to track the machine count. Four machines instead of two ought to mean double, and since the two numbers move together, the arithmetic feels safe. But every request still makes a trip to shared storage, and that storage did not multiply. Four machines often land near 1.7x and ten near 3x. Teams that miss this keep adding machines and end up paying five times as much for three times the speed.

The other frequent misreading is that horizontal scaling removes outages. Watching a machine die while the service keeps running makes that easy to believe. What actually happens is that the survivors absorb the dead machine's share — so a service barely holding on with four machines can lose one and then watch the remaining three fall over in sequence. The arrangement only pays off if the machine count leaves enough headroom to lose one.