Rolling Deployment

There was a time when deploying meant posting a maintenance notice. Take the servers down at 2 a.m., upload the new files, bring them back up, and the service is unavailable for thirty minutes with users warned in advance.

That pattern blocked several things at once. Somebody has to be awake at 2 a.m., so deployment becomes weekly, and changes pile up into each release. With many changes in one release, finding which line caused a problem is hard, and rolling back stretches the outage from thirty minutes to an hour. The next deployment then gets more cautious and the interval gets longer. Meetings kept producing the same sentence: let's hold what we didn't ship this week until the next maintenance window.

Rolling deployment solves this by using the fact that there are several servers. Instead of taking all ten down at once, replace two at a time across five rounds. While two are being replaced the other eight keep serving requests, so from the user's side there is no moment where nothing answers. No maintenance notice means deploying during the day, and deploying during the day means deploying several times a day.

It starts with deciding how many servers go in each batch. One at a time is the safest and slowest; half at a time is fast and spreads a problem to half your capacity. Two out of ten is a common choice.

1. Pull the first batch out of rotation. Stop the front layer from sending new requests to those two, and wait for requests already in flight to finish. Skip that wait and every connection open at that moment gets cut.
2. Swap in the new version. With containers — units that package everything needed to run so the software behaves the same anywhere — this step reduces to exchanging an image.
3. Confirm it's alive. Run a health check for whether the server answers at all, then add a smoke test, which checks quickly that the essentials work, to actually exercise a few paths like logging in and opening a document.
4. Put it back in rotation and watch for a few minutes. Compare error rate and response time against the previous batch. An error rate jumping from 0.2 percent to 3 percent stops the deployment here.
5. Move to the next batch and repeat from step 1. Ten servers means five rounds, and two minutes of observation per batch puts the whole thing around twenty minutes.
6. If something looks wrong mid-way, halt and roll back. Reverting the batches already swapped takes longer the further the deployment got.

Automation owns steps 1 through 5, and the humans write the stopping conditions in advance. Without a number — halt above 1 percent error rate, say — the tooling will see the anomaly and finish anyway, and the first person to notice will be a customer.

It fits where there are several servers and any one of them produces the same result. If a user's request is tied to state held on one specific server, taking that server down produces errors for that user alone. For a team deploying several times a day it's effectively the default.

The limit comes from the window where two versions run at once. Mid-procedure, four servers on the new version and six on the old are both taking requests, so anything where the two versions expect different contracts breaks in between. Adding a database column is the standard example. The old version reading a column only the new version knows about breaks, so the work splits: add the column first and get both versions tolerating it, then ship the version, then drop the old column. Skip that ordering and a zero-downtime deployment produces downtime, usually on whichever feature nobody thought to test against the old version.

Time is the other limit. Ten servers two at a time takes twenty minutes, and reverting takes roughly as long. During an active incident that twenty minutes feels long. Capacity deserves a look too: while two are out, the remaining eight carry ten servers' worth of traffic, so if normal load already runs above 80 percent, either use autoscaling to add machines for the duration or keep headroom in reserve.

"The rolling deploy is going two at a time and six servers are on the new version now."

Reporting that a deployment is in progress and how far it has gotten.

"This change drops a column, so it can't go out as a rolling deploy. It needs two stages."

Pointing out a change that can't survive the window where both versions run.

"Error rate spiked, so we halted at the third batch and rolled back."

A pre-agreed stopping condition fired and the deployment stopped.

At Pagedeck, Theo moved to this while eliminating the weekly 2 a.m. maintenance window. Twelve servers split into four batches of three, with three minutes of observation per batch, put one deployment at around fifteen minutes. Deployment frequency went from once a week to twice a day, and with fewer changes per release, tracing a cause got faster.

There was one incident in the first month. A change renaming a column the notification feature uses shipped in a single pass, and old-version servers looking for a name that no longer existed stopped notifications for eight minutes. Since then, changes to data structure ship as two deployments: one that adds, and a later one that drops the old column, with at least a day between them.

The most common misreading takes zero-downtime to mean users notice nothing. The service not stopping is accurate. But connections held on a server being replaced do get closed, and while the deployment runs two versions are live, so a refresh can land on either one. That's why users see the old layout and the new layout alternate during a release that changed the layout. Zero-downtime means nothing stops, not that everything looks identical.

Believing a problem can be undone quickly is the other one. Changing gradually makes it feel like undoing is gradual too. But reverting walks the same procedure backwards, so it costs whatever the deployment already spent. Rolling back from eight replaced servers means four more rounds. For a service that has to return to the previous state within seconds, standing up a whole separate environment and switching once is the right shape instead.