Infrastructure

There was a time when putting a service on the internet meant buying the machines first. Getting one server took a quote, a purchase approval, and a wait for delivery. When the box arrived it had to go into a building with power and a network line, in rented rack space. From the day of the order to the day something ran on it, a month was normal.

The price was the smaller annoyance. A team had to decide how many machines to buy before it knew how many people would show up, and an error in either direction cost money. Buy too few and the site fell over on the day an ad ran. Buy comfortably and eight machines out of ten sat there drawing power without doing anything. Sizing for the busiest day of the year and letting the hardware idle through the other 364 was simply how it was done.

More machines brought a second problem. Servers installed by hand were never quite identical. Server 1 carried a setting someone had edited by hand during an incident, server 5 did not, and so the same code ran correctly on server 1 and failed on server 5.

Conversations in an engineering team of that era had a shape. "Can we get one more server?" was answered with "the quarter's budget is gone, I'll file the request next month." A feature took a day to build, and the place to put it appeared six weeks later.

Infrastructure is the name for that floor, once people pulled it out and treated it separately. Whatever gets built on top, something underneath has to do the arithmetic, carry the requests, and keep the records. A feature is finished when it ships; the floor is held for as long as the service is alive. A floor that is plenty for ten users collapses in front of a hundred thousand, and a team that has not decided in advance how it grows and shrinks finds the service stopping on its best day.

The terms under infrastructure fall into six branches. Where you get the floor from is form. What you put the running program into is runtime. How a request from outside reaches it is network. How you enlarge the floor when a crowd arrives is scaling. The last two are where the setup is written down, which is management, and the delay it creates on a first request, which is performance.

Form divides how machines are obtained. Cloud means renting as much as you need from a data center somebody else built and paying for what you used. The six weeks from the previous section become a few minutes here. You choose a size on a screen, press create, and two minutes later there is a machine to work with; delete it and the charges stop at that moment. In exchange, a cost once paid a single time becomes a monthly one, so a machine left running and forgotten keeps drawing money quietly.

Runtime divides what a program is packed into once the space is rented. There are three, arranged in order of getting lighter.

1. Virtual machine — one physical server split so it behaves like several. Each one carries a whole operating system of its own, so whatever happens inside one has no effect on the room next door. The cost is that a full operating system has to boot, which takes one to two minutes and claims memory up front.
2. Container — the program and the pieces it needs to run, bundled together, with the operating system underneath shared rather than duplicated. That is why a container starts in seconds and dozens fit on a single machine. Because the bundle is fixed, what ran on a laptop runs the same way on the server. The difference between server 1 and server 5 from the previous section disappears here.
3. Orchestration — once containers number in the hundreds, nobody can track which machine holds how many, or where to restart one that died. Orchestration hands that judgment to software. Write down the state you want, as in "three copies of this program are always running," and the moment one dies a replacement is started to restore that state.

Network divides the path a request from outside takes to reach the program. Five terms sit along it in the order the request meets them. Type a name into the address bar and the domain name system (DNS) turns that name into the real address of a server. It is a phone book pairing names with numbers, so a server can move and the name people type stays the same once the entry is updated. The request that found the address arrives at a load balancer, which hands it to whichever machine behind it has room right now and drops one that has stopped answering. That is usually the reason a service keeps running when a single machine dies. A reverse proxy receives requests on behalf of those machines and passes them back, handling shared work such as compression and certificates in one place. Files that go out identically to everyone, such as images, are handled by a content delivery network (CDN), which copies them to points around the world in advance and serves each visitor from the nearest. A call from Seoul to a server in the United States spends about 200 milliseconds on the round trip; the same file from a point inside Seoul takes 10. Edge computing goes one step further and runs small calculations at those nearby points.

Scaling divides how the floor is made bigger when a crowd arrives. There are two directions, with one piece of automation sitting on top of them.

1. Horizontal scaling — add more machines of the same size, going from 2 to 10. There is almost no ceiling, and the loss of one machine leaves the rest working. In exchange, the program has to be written so that any machine produces the same result.
2. Vertical scaling — make one machine more powerful, moving its memory from 8 gigabytes to 32. No code has to change, which makes it the fastest move available in an emergency. The limits are that a single machine can only grow so far and that it stops while the change is applied.
3. Autoscaling — hand the decision to add or remove machines over to a rule. Write "if load stays above 70 percent for five minutes add one machine, and if it drops below 30 percent remove one," and the service runs on two machines at dawn and eight at lunchtime. It works while everyone is asleep, and the money spent on idle machines falls with it.

Management is where all of this configuration is recorded. Infrastructure as code (IaC) means writing the number of servers, their sizes, and their network settings into a file and running that file, instead of clicking them into existence on a screen. Once written down, the same file builds development and production environments that match, a record shows who changed what and when, and the whole setup can be rebuilt from the file if it is ever wiped. A configuration assembled by clicking exists only in the memory of the person who clicked.

Performance covers the delays this kind of floor introduces. A cold start is the first request being noticeably slow because a runtime that had been idle has to wake up. Switching things off while nobody is using them saves money, but the person who arrives first afterwards waits three seconds for a screen that normally takes 0.1. A decision made to cut costs lands on one user's first impression, so teams separate the places worth putting to sleep from the places worth keeping warm.

The six branches are not a flat list; each one called the next into being. Machines that could be created in minutes made stamping out identical copies useful. Many copies gave a reason to put something in front of them to divide the traffic. Divided traffic made changing the machine count automatically possible. And once the moving parts passed what a person could hold in memory, writing the configuration down as code followed.

"Traffic is going to spike for Friday's event, so I'll scale the infrastructure up beforehand. I'll bring it back down when it's over."

The team knows the usual number of machines will not hold, so it is growing the floor in advance. The promise to shrink it again is there because everything added keeps costing money until it is removed.

"That isn't a code problem, it's an infrastructure problem. One of the three machines behind it had stopped answering."

This explains a symptom where the same request worked for some people and failed for others. No amount of reading the code finds this class of problem, so the speaker is proposing to look elsewhere.

"Infrastructure costs went up 40 percent from last month and the user count is flat. Let's find where it's leaking."

When you pay for what you use, the bill can climb while the number of users stands still. The cause is usually a machine created for a test and never deleted, or a scaling rule written with the wrong threshold.

At Pagedeck this word came up constantly in the month after one enterprise customer signed. Until then the product had run on two servers, at dawn and at lunchtime alike. When 300 people at the customer arrived at work and opened documents within the same few minutes, two servers could not carry it, and support tickets saying a document took 8 seconds to open came in all morning.

Theo put the fire out with vertical scaling. He doubled the size of both machines that evening, changed not one line of code, and got the team through the next morning. The real response came after that. Hourly access records showed the window from 9 to 11 in the morning running four times heavier than the rest of the day, so he set a rule that grew the service to six machines on weekday mornings and let it fall back to two overnight. Grace asked one question: how much the monthly bill would go up. The answer was 18 percent.

The first mistake is assuming infrastructure belongs to whoever is assigned to it, and that a developer only has to write code. Large companies really do have a team that does nothing else, and even in small teams the permission to log into a server often sits with one or two people, so the impression forms easily. But the shape of the code decides what is possible on the floor underneath it. Keep a signed-in user's session in the memory of one server and that person has to keep landing on that same server no matter how many machines you add, which closes off horizontal scaling. Design without knowing this, and when traffic grows the question "can't we just add servers?" gets the answer "not until we change the code," and a day of work turns into two weeks.

The second is expecting a move to the cloud to make infrastructure somebody else's concern. Nobody buys hardware and nobody visits a data center, so the whole area feels deleted. What was deleted is the work of obtaining machines; the number of decisions actually went up. Which region to run in, how many machines to start with, when to grow and shrink, where backups go and how many days to keep are all choices now, and a choice not made is made by a default. Those defaults lean generous, so the cost shows up first. Leave this area blank for six months and you find out about twelve machines nobody is using when the invoice arrives.