Cold Start

Back when you bought a server and left it running around the clock, this delay didn't exist. The program came up once and stayed up, and the first request got the same speed as the ten-thousandth. The cost was that the machine also ran at 3am with nobody using it, billing the same amount. A service that is genuinely busy for two or three hours a day was paying for the other twenty.

Cloud billing by actual usage opened a way out. Shut the environment down while no requests are coming in and those hours cost nothing, because you are charged for execution rather than for owning a machine. For some services the server bill dropped from around $1,500 a month to $150, which for an early company is the difference between one more month of runway and not having it.

That came with a price. Turning something back on takes time. Infrastructure divides into where the underlying machines come from, what you package the program into, how requests travel in, how you grow capacity under load, and where the configuration is written down. Cold start belongs to the performance side — the first delay that the chosen arrangement produces.

In practice it usually looks like a bug. Theo clicked login before a demo and it took four seconds. He clicked again and it came back in 0.2 seconds. It wouldn't reproduce, and then it was slow again on the first click after lunch. No code had changed and the servers looked healthy, so without a name for it you can burn a day looking.

When a request arrives at a slot that is powered down, four things happen in order before your code runs at all.

1. Getting a slot. The platform decides where to place the machine or the container — a unit that bundles the program with everything it needs to run — and reserves that space.
2. Starting the runtime. The language runtime comes up in that slot. Runtimes with a lot of startup work, like Java or .NET, can spend more than a second right here.
3. Loading and initializing. Your code and the external libraries it depends on get read in, then setup work runs: opening database connections, reading configuration.
4. Doing the actual work. Only now does the request get handled, and this part is the response time you normally see on a healthy system.

Steps 1 through 3 are the cold start. A light function spends 0.2 seconds there; a Java service pulling in many libraries can spend three to five. Once the same code is awake it answers in 20 to 50 milliseconds, so the gap runs from ten times to a hundred times.

An environment that has woken up doesn't shut down immediately — it stays alive for minutes or tens of minutes, and every request in that window is fast. The trouble comes with the first request after the window closes, and with bursts that the live slots can't absorb. If 100 people arrive at once and three slots are warm, the other 97 each pay for a cold start.

There are three ways to shrink it. Lightening step 3 means stripping unused libraries so the deployment bundle is smaller. Avoiding step 2 means choosing a runtime that starts quickly. The last is refusing to let it shut down at all: configure a minimum number of always-on instances and the cost comes back in proportion. Treat it as a dial between savings and response time.

The most common place is serverless. Code uploaded as functions and run only when a request arrives is, by design, shut down the rest of the time, and that property turns directly into cold starts.

Autoscaling produces the same effect. It adds and removes servers automatically as load changes, but once it decides to add one, that server needs 30 seconds to two minutes to be ready, and requests arriving meanwhile pile onto the existing machines. That is why the rule usually triggers at 60 percent load rather than 80 — the head start covers the gap.

The phrase gets used about caching too. Caching stores a result so it can be reused, so right after the store is cleared and restarted, every request has to travel to the original source. The few slow minutes after a deploy are usually this.

How much any of it matters depends on the spot. A billing job that runs overnight can start three seconds late and nobody notices. An API — the agreed interface programs use to talk to each other — sitting behind a payment button that stalls for four seconds gets read as a failed payment, and the user clicks again.

"This API has a three-second cold start, so we can't put it in the payment flow."

An objection to using this arrangement anywhere a person is sitting and waiting for the screen to respond.

"Reports say only the first request each morning is slow. Looks like a cold start, let me check the logs."

No traffic overnight means the environment shut down, and the first caller pays for the wake-up.

"Let's keep two minimum instances warm. Sixty dollars a month buys away the three-second first response."

A proposal to trade some of the savings back for a predictable first response.

At Pagedeck, file preview moved to serverless last quarter. Average response time actually improved, to 180 milliseconds, and support tickets went up anyway. Theo found that the average was made of the many requests served by warm slots, while a few dozen requests a day took four seconds and disappeared into that average.

So he redrew response time against the slowest 1 percent instead of the mean, and the four seconds stood out immediately. Keeping two minimum instances warm fixed it, at about $60 a month.

It's easy to assume optimizing the code will make it go away. Slow responses have usually been a code problem, so the guess is reasonable. But much of a cold start is time that has already passed before your code runs. Polishing the code leaves the runtime startup exactly where it was. The dials that actually move it are bundle size, runtime choice, and how many instances you keep warm.

Watching only the average response time is the other common miss. Cold starts hit 1 to 5 percent of requests, so folding them into a mean produces a perfectly healthy-looking 180 milliseconds. Left that way, the dashboard stays green while the tickets keep arriving. Chart the slowest 1 percent and 5 percent separately and it shows up on sight.