Percentage Rollout

For a long time a release had only two states: not out, or out to everyone. The moment a feature was switched on, every user saw it, which meant that any defect in it also reached every user at the same moment.

Release evenings all looked alike. The new editor went live at six on a Friday, support tickets started arriving at seven, by nine someone had worked out which browser was failing to save, and a rollback went out in the early hours. The old version came back, but nobody could say how many thousands of people had failed to save a document in between.

What the team took away was that it should test more, and testing was not where the defect lived. Some failures only appear under real traffic: the point where too many concurrent users overwhelm a server, or a data shape that survives only in accounts created years ago.

A percentage rollout moves that check to after the deploy. It assumes a defect is still in there somewhere and tries to find it while keeping the number of people who run into it as small as possible.

This is the stage that follows a canary release, where the feature opens to the smallest group first. Once that group looks clean, the share goes up in steps — six of them.

1. Plant the switch. A device that turns a feature on and off while the code stays deployed is a feature flag, and every change to the percentage happens on top of that switch.
2. Decide how people get assigned. Hashing the user ID is the usual approach because it keeps the same person on the same side. Rolling the dice per request instead means one user flips between the new screen and the old one, and the support tickets start before the data does.
3. Write down the numbers you will watch and the line that stops the rollout. Pick the measures first — error rate, response time, save success rate — then put the threshold in figures, as in "stop if the error rate runs 0.5 percentage points above the control group."
4. Raise the share. A common ladder is 1, 5, 25, 50, 100, with at least a few hours and often a full day on each rung. A screen seen by 300 people at 1 percent is seen by 7,500 at 25 percent.
5. Compare the two groups at every rung. Put the group with the feature on next to the group with it off, over the same hours. How much you can tell from outside about what is happening inside a system is called observability.
6. Clean up when you reach 100 percent. Strip the branch and the switch out of the code and delete the old path.

If the numbers go bad partway up, drop the share to zero. The device that kills just the feature is a kill switch, and because it needs no new deploy it takes seconds. You turn it off and then look for the cause, not the other way around.

It fits changes that are cheap to undo. Swapping one screen or adding one feature goes back to where it started as soon as the switch comes down.

It needs enough users to work at all. How many people 1 percent is decides whether the method does anything. With 30,000 monthly users, 1 percent is 300 people, and an error anyone would notice shows up in that group. In a service with 5,000 users, 1 percent is 50 people, so a rise in the error rate amounts to a single incident and signal can't be separated from noise.

Changes to how data is stored need more than this on its own. The new storage shape and the old one have to run side by side for a while, and dropping the share back to zero leaves the question of what to do with the records already written the new way. Put a dark launch in front of it — running only the new path behind the scenes, with nothing visible to users — and compare the two before raising any percentage.

The fact that users see different screens is a limit too. In a collaborative document tool, one person on the new sharing screen and a teammate on the old one will describe the product differently to each other, and support volume goes up. Features like that get assigned by team rather than by individual.

It also isn't a way to judge whether a metric improved. An experiment that splits users at random between two versions and compares the results is an A/B test; the groups in a percentage rollout change size at every rung, so they can't carry that comparison. What you are watching here is whether anything broke.

"We're at 5 percent and the error rate is flat. I'll take it to 25 this afternoon."

The previous rung met its criteria, so the next one is going ahead.

"Save failures are up at 25. Let's drop to zero and then look at why."

Kill the feature now, investigate second.

"Let's hold this one at 50 for two days. I don't think we'll know until month-end traffic goes through."

Cover the period when load spikes before calling it safe.

That is how Pagedeck opened its new editor. Theo planted the switch, deployed, and opened 1 percent so 400 people saw it; once the error rate looked identical he moved to 5 percent. The first real problem turned up at 25 percent: accounts holding more than a hundred documents took three times as long to load their list. Accounts like that are under 1 percent of the base, so not one of them had landed in an earlier rung.

Mia dropped the share to zero on the spot, and Theo had a fix out three days later and restarted from 25. Opening it to everyone at once would have handed the same slow list to every document-heavy account on the same day.

The most common belief is that dividing a release into percentages is itself the safety measure. Having rungs makes it look as though something is now protecting you. But the value of the rungs comes from reading numbers between them. With no measures chosen and nobody watching, 1 percent is as blind as 50 percent: you find out when tickets arrive, exactly as before, except the schedule is now days longer.

Reading 100 percent as the finish line is the other one. It is finished for users, but the branch is still sitting in the code. Leave it there and the next feature has to account for the old path too, and in a codebase carrying twenty stale switches nobody can say which are still live.