Rollback

For a long time, learning that deployed code was broken left one path: find the cause, fix it, deploy again. The problem is the time that takes. When the cause doesn't turn up, one hour becomes two, and users keep looking at a broken screen the whole time.

Pagedeck lived through this. Theo shipped a change to how documents save, and after that roughly one save in twenty failed. The screen said saved while nothing had been written, so support tickets arrived quickly. Theo spent forty minutes hunting for the cause and couldn't reproduce it, because the failure only appeared under one particular condition. For those forty minutes, people writing documents kept losing them.

Rollback reverses the order. Recovery gets separated from diagnosis, and recovery goes first. Put yesterday's code back in place and the damage stops right there. Why it happened is a question you work through while users are looking at a working screen.

The criteria for reverting get written before the release, and when a number crosses one of them, you execute without stopping to deliberate. Inventing the criteria at 3 a.m. is already too late. Deliberation is the slow part, so it gets moved to before the release.

1. Fix the thresholds and the authority in advance. Write them as numbers: error rate above three times normal, save success rate below 99 percent. Write down who is allowed to revert, too. Without that, the night it matters gets spent looking for whoever decides. A threshold nobody wrote down turns into an argument at the worst possible hour.
2. Check whether something lighter will do. If the damage is confined to one feature, a kill switch turns off that feature alone. The switch that turns a feature on and off while its code stays deployed is a feature flag, and a kill switch is what drops that flag in a hurry. If the flag settles it, nothing needs reverting.
3. Name the version you're going back to. The release immediately before is the default, and if that one is also suspect, you look one further back. Teams that don't keep a record per release get stuck at this step.
4. Revert, then verify. Verification isn't opening the screen — it's watching the numbers from step 1 return to where they were. If they don't, this release wasn't the cause, and the work moves into incident management, the whole practice of recovering from a failure and keeping it from recurring.
5. Deal with the data written in the meantime. Old code may not be able to read records the new code created, so reverting is often followed by migrating or deleting those records. This step is most of what makes a rollback heavy. A thousand documents already saved in the new format still have to open after the revert.
6. Write it down. When, what, why, and the condition for shipping it again. Without that line, the same code goes back out two days later.

Three situations suit it: the deployed code itself is the cause, the cause is still unknown, or a lot of users are affected. The last one is about cost: when work is being lost rather than merely delayed, waiting for a diagnosis is the expensive option. The second matters most. Fixing forward without knowing the cause is a gamble, while reverting goes to a state that worked yesterday, so the result is predictable.

Four limits.

1. Releases that changed the data shape — once a new column is collecting values, reverting the code leaves those values in place. If the old code doesn't know the column, it fails again on read. So a release that changes structure gets its reverse path built alongside it.
2. Anything already sent outward — mail that went out, a payment that was approved, an order handed to an external system. Reverting the code doesn't recall any of it. The revert makes the next message correct and does nothing about the ones already delivered.
3. Small problems — wrong wording on a button doesn't justify reverting everything. Fixing that one piece and shipping immediately is faster.
4. Nothing to go back to — a feature shipping for the first time has no previous version. Turning off its flag is the only route back to the state before it, which is why new features ship with a flag in place.

Teams that open to a tiny group first with a canary release, then raise the share with a percentage rollout, rarely reach for a rollback. A problem caught in the 1 percent window ends with the flag coming down.

"Let's roll back first and find the cause afterward."

Putting recovery ahead of diagnosis.

"A rollback won't fix this. Data is already in the new column."

Reverting the code alone leaves old code failing on new records.

"We agreed that below 99 percent save success we revert without asking."

Confirming that thresholds and authority were settled before the release.

Since that saving incident, Pagedeck puts one revert line in every release note. Theo writes which number, falling how far, triggers a revert, and Mia checks that line before approving the release.

When the same saving change went out two months later, the failures were spotted within ten minutes and the revert took three. The cause turned up the following week — it only occurred in one browser — and the third attempt, shipped with that condition understood, held.

The most common reading is that a rollback is a record of the team's failure. A revert leaves a trace, so it reads that way. But being able to revert is evidence of preparation. Treat reverting as a black mark and the team chooses to patch forward in a hurry instead, adding a second piece of under-tested code in the middle of an outage. The number worth counting is not how often you reverted but how long it took to notice a problem and get back to normal. A team that reverts twice a month and recovers in four minutes is in better shape than one that never reverts and takes two hours.

The other common belief is that you can always revert. Deployment tools with a single button make it feel that way. Code goes backward, though, and data and anything sent outward only go forward. Treated as unlimited insurance, reverting lets a structural release ship with no reverse path, and the moment it's needed, fixing forward is the only option left.