Mean Time to Recovery

For a long time, nothing numeric survived an outage. What survived was an impression. Someone would say things seemed to be breaking a lot lately, and engineering would answer that this one came back faster than the last one. Neither side had evidence, so the conversation ended in the same place every quarter, and nobody left it knowing what to fix next.

Quarterly reviews were where this stung most. A founder would ask why last month's outage dragged on, and the answer was usually that it took a while to find the cause. Whether "a while" meant thirty minutes or six hours was something no one in the room could say. The record of the incident lived scattered across chat messages, and lining up when it started against when it ended meant someone spending half a day reading logs backward.

MTTR was borrowed to replace that exchange with a number. It started life on factory floors, where maintenance teams measured the average minutes needed to get a stopped machine running again and used that figure to decide where to station technicians. Software operations, where a stopped service means stopped revenue, took the idea as it was. What gets measured here is the incident — an event that stopped users from doing what they came to do. Not every error line in a log counts, only the ones with confirmed user impact.

Measure the time each incident took to resolve, add those times together across a fixed window, and divide by the number of incidents.

The calculation has three steps.

1. Fix a start time and an end time for each incident. The start is when user impact began; the end is when the service was working normally again.
2. Work out the duration of each one. Three incidents in a month lasting 20, 40 and 180 minutes add up to 240 minutes.
3. Divide by the count. Dividing 240 by three gives an MTTR of 80 minutes for the month.

Step 1 is the one teams argue about, specifically the start time. A team has to decide up front whether the clock starts when the failure began or when an alert fired and somebody noticed. Starting the clock at detection produces a backwards result, where the slower a team is to notice, the better its number looks. An outage that began at 2 a.m., was found at 9 a.m. and was fixed by 9:30 runs seven and a half hours on the first definition and thirty minutes on the second. Users lived through seven and a half hours.

Settle the window in advance too. Most teams report monthly or quarterly, and with few incidents the average swings hard. Three incidents at 20, 25 and 30 minutes average out to 25; add one eight-hour incident and the average jumps to 141. That is why the average rarely travels alone. Teams publish the median and the longest single incident beside it, because the average gets dragged around by one bad night while the median shows what an ordinary incident looked like.

There is no pass mark that holds across services. A checkout flow that leaks money after two minutes of downtime and a nightly settlement batch cannot be judged on the same scale. The bar comes down from the service level agreement (SLA), the service quality a company has contractually promised its customers. Promise 99.9 percent monthly uptime and the budget for being down is 43 minutes; drag a single incident out for an hour and the promise is broken for that month.

As a rough gauge, teams with settled operations tend to close incidents inside an hour, and recovery that stretches past a day usually means a step in the process is missing rather than a hard bug. The DORA metrics, a set of four measures for delivery performance, include time to restore for the same reason and put the top group under an hour.

What actually reads well here is the trend, not the absolute figure. Four hours falling to forty minutes over two quarters says the on-call rotation and the recovery playbook have settled into place. The way to get there runs through finding the cause later rather than faster. A rollback to the previous version brings the service back first, and the investigation happens afterward on a system that is serving users again. Teams that insist on diagnosing before fixing spend every minute of that diagnosis with customers locked out.

It is also worth checking the on-call rotation for hours nobody covers. A team where a Friday night failure gets found on Monday morning will post a bad number no matter how good its engineers are at fixing things.

"Our MTTR went from two hours forty to fifty-five minutes this quarter."

Not a claim that fewer things broke — a claim that each break took less time to clear.

"Detection was late on that one, so it pushed the whole MTTR out."

The fix was quick; the time went into noticing there was anything to fix.

"If we set an MTTR target of thirty minutes, people will start skipping the analysis."

Naming a side effect of the number before someone turns it into a goal.

At Pagedeck, Theo brings this figure to the monthly operations review. Last month had five incidents and an average of 90 minutes, and next to that average Theo writes the median of 22 minutes and the longest incident at six hours. The six-hour one is what pulled the average up, and the fix itself took 25 minutes. The other five and a half hours were time nobody knew anything was wrong.

The question Grace asks is not why the number is 90. It is why no alert fired during those five and a half hours. So the work that follows is not about recovering faster — it is about fixing the alerting rules.

The most common misreading is that a low number means a stable service. Faster recovery means shorter pain for users, so the link feels obvious. But this figure says nothing at all about how often things break. A service that falls over three times a day and comes back in ten minutes each time has an MTTR of ten minutes, which beats a service that breaks once a quarter and takes half a day. Judge operations on this number alone and the team gets very good at putting out fires while the question of why fires keep starting slides into next quarter, every quarter.

Setting it as a target is the other standard mistake. Turn a measurement into a goal and people move toward whatever makes the goal easy. A team handed a thirty-minute target starts skipping the postmortem and declaring incidents closed early, or stops counting small events as incidents at all. Nudge the severity threshold up a notch and the short incidents drop off the list, which pulls the average down on its own. Six months later the number reads thirty minutes and the hours customers spend locked out have not moved.