Kill Switch

When a newly released feature started misbehaving, the only available move used to be fixing it. Find the cause, change the code, ship it again — and a single deploy takes anywhere from twenty minutes to an hour. Add the time spent finding the cause and two or three hours pass between noticing the problem and stopping it, with users on that screen the whole time.

That is roughly what happened the night Pagedeck opened its new editor to everyone. At two in the morning the save failure rate jumped to twenty times its normal level and Theo, who was on call, woke up and started reading logs. Working out that the failure only occurred in one browser took forty minutes. Fixing it and deploying took another thirty. During that hour and ten minutes, several hundred documents failed to save.

In hindsight the sequence was backwards. The whole approach assumed you have to understand a problem before you can respond to it, but at two in the morning the urgent thing wasn't the diagnosis — it was stopping the bleeding. The old editor was still running perfectly well, and sending users back to it would have dropped the save failures to zero on the spot.

A kill switch inverts that order: turn it off first, investigate second. What the situation calls for at 2am is one switch, not a deploy pipeline.

The core job is separating the time it takes to stop the damage from the time it takes to fix the problem. The fix can take two days as long as stopping takes a few seconds.

Three distinct delays get cut out.

1. Diagnosis time — digging through logs to learn which conditions trigger the failure. Deciding to turn something off doesn't require that answer. Knowing that the numbers are bad only in the group with the feature enabled is enough, and that comparison is usually available within a minute of the alert.
2. Ship time — editing code, passing checks, deploying. A kill switch changes one configuration value on code that is already deployed, so it skips this stage entirely.
3. Decision time — if nobody has been told they're allowed to pull it, half an hour disappears into finding someone to call at 2am. So the authority gets written down alongside the switch itself: who can flip it, under what conditions. Usually the on-call engineer gets to pull it without asking anyone first.

Some teams encode the decision as a number, wiring the feature to shut itself off when the error rate crosses a threshold without a human in the loop. That requires observability first — the degree to which you can tell from the outside what's happening inside the system. Without it, a team sleeps through the whole incident and finds out in the morning.

The other thing to prepare while building the switch is the state it leaves behind. What the user sees once the feature is off has to be a decided question. Turning off the new editor puts the old editor back, which only works because the old path still exists. Delete the old path and flipping the switch produces an empty screen, which is a worse outcome than the failure it was meant to stop.

When turning the feature off doesn't fix the situation, three heavier moves are available.

1. Lowering exposure — if a percentage rollout has reached 25 percent, dialing back to 1 percent is an option instead of going to zero. It suits cases where the team wants to keep observing the behavior on a small group.
2. Rollback — when flipping the switch doesn't help, the deployed code itself goes back to the previous version. This is the move when something shared broke rather than the new feature.
3. A maintenance page — closing the whole service and showing a notice. Reserved for cases where data keeps getting corrupted and letting people continue is worse than the outage, since every minute of a maintenance page is visible to every customer at once.

There's also the option of needing the switch less often. A canary release opens a feature to a very small group first — one percent of traffic, or internal accounts — and watches whether their numbers diverge. A problem caught there never becomes a 2am switch flip at all. A kill switch is the last line of defense, not the first.

"Pull the kill switch now, we'll look at the cause in the morning."

Stopping the damage comes first; the diagnosis can wait.

"If this feature has no kill switch it isn't going out this week."

Something that can't be turned off doesn't meet the bar for release.

"Let's let on-call pull it without an approval."

Don't spend the night hunting for a decision-maker.

After the editor incident, Pagedeck adopted a rule: any feature that changes what users see has to have an off switch to pass release review. Mia added a line to the release checklist — how it gets turned off, and what appears once it is — and Theo wrote the shutdown procedure on a single page so whoever is on call can follow it half asleep.

Four months later something similar happened again. This time the alert fired and the feature was off four minutes later, and the cause was found the next day during working hours. Twelve documents were affected.

It's easy to assume that turning a feature off undoes it. The feature disappears from the screen, so it looks that way. But only future behavior stops; whatever happened while it was on stays. Notification emails sent by mistake are already sitting in people's inboxes, and data written in the new format may be piling up in a shape the old path can't read. That's why the design of a switch includes what remains after it's pulled. Skip that question and turning the feature off just leads into a larger problem.

The other common mistake is thinking the work ends once the switch is built. A switch feels permanent, but it quietly breaks as the code around it changes — somebody cleans up a branch and removes one side of it, the old path disappears along with an unrelated change, or the value in the settings screen stops being connected to anything. A switch nobody has ever pulled does not work on the night it's needed. Teams that actually rely on theirs pull it in a staging environment now and then and check that the fallback screen still renders.