Time to Restore Service

An instruction to reduce incidents usually drifts into reducing deployments. Nothing breaks on a day nothing ships, so the count really does fall. But no organization reaches zero incidents, and a team that ships less puts more into each release, which makes any one incident last longer. Fewer and larger releases also make the cause harder to isolate when something does break.

That played out over a full quarter at Pagedeck. Grace asked for fewer incidents, and deployments went from five a week to one. Quarterly incidents dropped from three to two. One of those two ran for four hours. Support questions over the same period went up rather than down. What users experienced was not a count but the time the service was unusable.

Time to restore changes the question. Not how often things break, but how fast they come back once they do. Preventing breakage and recovering quickly are separate investments, and the second one is the more reliably controllable of the two. The number describes a response, not a run of luck.

Measure the gap between when the incident started and when the service returned to normal, then take the median across the incidents in a period. Five things get decided.

1. The start is when users began to be affected, not when you deployed. If 35 minutes passed with nobody noticing, those 35 minutes are in. Start the clock at detection instead and the problem of detecting late disappears from the number. A gap between impact and detection is itself a finding worth recording.
2. The end is when the impact stopped, not when the cause was fixed. If a rollback to the previous version brought screens back, the count stops there. The root-cause investigation continues afterward and doesn't enter this value. Each incident therefore carries two timestamps, impact ended and cause fixed, and only the first belongs here.
3. What gets counted is an incident that affected users. Leave out an internal tool blinking or an alert that fired with no impact — and write down in advance what's in and what's out. That list of exclusions is what keeps the number comparable from quarter to quarter.
4. Use the median, not the mean. Four incidents at 10, 15, 20 and 240 minutes average to 71 minutes and have a median around 17. The mean lets one unusual case hide the other three. Reporting the median and the maximum together keeps the bad case visible without letting it set the headline.
5. Don't leave the recording to people. Have the tooling capture when the alert fired and when resolution was declared. Filled in later, the times get remembered shorter than they were. A resolution declared in a chat channel is enough, as long as the tool stamps it.

With only four incidents in a quarter, one case moves the median a lot. Grouping by quarter or half-year to read the trend works better than reading it monthly.

Your own trend matters more than a figure borrowed from outside. Thirty minutes last quarter and 18 this quarter means something improved; a verdict drawn from an industry average compares organizations with different system shapes. For reference, under an hour is respectable, and repeated incidents running past a day mean there's clearly something to fix.

Three things produce this value.

1. Detection. How much you can tell from outside about what's happening inside is observability, and when it's low, half the number is filled with time nobody knew. Alerts tied to user-visible symptoms shorten this stretch more than alerts tied to server metrics.
2. Response start. The person scheduled to take the call is on-call. With no rotation, an incident at 3am runs until morning.
3. Reversal. Whether one action returns the system to its previous state sets the final stretch. A deploy reversible by one command is worth more here than any amount of prevention work.

Start from the third if you want the number down. With reversal automated, the impact can end before anyone knows the cause. That's why teams shipping in small pieces also recover faster: a high deployment frequency means each release is small, so reversing one costs little.

Read alone it invites misreading. Put it beside change failure rate, the share of deployments that caused an incident. A 15-minute recovery still means users hit that 15 minutes often if one deployment in five breaks something. What users experience is the product of the two, not either one alone.

"Median is 17 minutes this quarter. I'll handle the 240-minute one separately."

Separating the trend from a single exception.

"Detection took 35 minutes. The fix took five — the problem is that no alert fired."

Identifying which stretch inside the same number ran long.

"We still don't know the cause, but the rollback ended the impact. That's where we stop counting."

Confirming that this metric measures impact, not diagnosis.

Going through the quarter's four incidents, Theo found only the 240-minute one stood out. That release had shipped a change to storage structure alongside the code, and with the structure already converted there was nothing to roll back to. The other three all ended inside 20 minutes by reverting.

Since then Pagedeck ships storage changes and code changes on different days. The structural change goes out first in a reversible form, and the code follows once it has settled. The next quarter's maximum was 40 minutes. The maximum falling before the median did was the effect of that change. The lesson was about how releases are composed rather than about response speed.

The most common reading is that this measures how quickly a team can fix things, because "restore" suggests repair. What it measures is the end of impact, not the removal of the cause. In practice most incidents end by reverting with the cause still unknown. Make root-cause analysis a condition of recovery and users spend longer looking at a broken screen — time that comes back as this number. The two clocks get conflated in postmortems more than anywhere else.

Reading a short number as stability is the other one, since one good figure makes an area feel healthy. But this value assumes an incident happened. Without frequency beside it, a team that breaks three times a day and recovers quickly three times a day scores as excellent. Frequency and recovery also move in opposite directions under the same pressure. That's the reason the four metrics are kept together.