Mean Time Between Failures

How often things break was, for a long time, a matter of impression. Engineering would say the service had been pretty stable lately and sales would answer that a customer had called about an outage just last week. Neither was lying. One side remembered that nothing major had happened, the other remembered the two times their own accounts were affected.

Faster recovery widened the gap. An outage used to take half a day, so everybody remembered it. Once rollback procedures settled into place, plenty of failures resolved in three minutes — and a three-minute interruption leaves no trace in anyone's memory. A service can go down twelve times in a month and still look like a quiet month from the inside, while for the customer who happened to hit save at that moment it was twelve incidents.

Pagedeck ran into exactly this. In a monthly review Theo said the month had passed without a major outage, and Mia said document saving had failed on her three times. They remembered the same month differently because one was counting the size of failures and the other was counting how many there were.

Mean time between failures (MTBF) pins down the second of those. It has nothing to say about how bad an outage was, only about how often one happened. "Things feel stable lately" turns into "the gap between failures went from 60 hours to 240."

Add up all the time the service was running normally and divide by the number of failures in that window. Over 30 days (720 hours) with three failures and six hours of total downtime, 714 healthy hours divided by three gives 238 hours — something broke roughly every ten days.

There are four steps.

1. Pick the window. A month is the default; services that fail rarely use a quarter. A short window swings hard on a single incident.
2. Decide what counts. Only events judged to be an incident go in — the concept that separates the errors piling up in server logs from the ones that stopped a user doing what they came to do.
3. Work out healthy running time by subtracting downtime from the total.
4. Divide that by the number of failures.

Step 2 determines almost everything about the resulting number. How far down the severity scale you count changes the same month's value dramatically. Count only severity 1, where every user lost the ability to save, and the month had one failure and an MTBF of 714 hours. Count down to severity 3, where text rendered wrong in one browser, and it had nine failures and an MTBF of 79 hours.

So the counting rule gets written down first, and values from either side of a rule change don't get joined into one line. When the number halves in the month a team widened its definition, the service didn't get worse — the ruler changed.

There's no absolute benchmark. Whether 240 hours is good depends on what the service does. For a payment system it's alarming; for an internal report generator it's generous. The only fair comparison is against your own past, measured the same way.

The direction is one-way: this number should grow. Going from 60 hours to 240 means the gap widened fourfold, and the only thing that produces that is improvement items from root cause analysis actually getting done. When a full disk brings the service down, clearing the disk is recovery; wiring up alerting so someone hears about it while space is still shrinking is what moves this number.

Read it alongside MTTR, the average time it takes to recover from a failure. Watch that one alone and a team gets very good at putting out fires while the fires keep starting. Put the two together and you get uptime: with an MTBF of 720 hours and an MTTR of one hour, 720 divided by 721 is 99.86 percent. Doubling the gap and halving the recovery move uptime by the same amount.

Small sample sizes deserve caution too. With two failures in a month, one more or one fewer halves or doubles the value. Going from two to three drops it from 357 hours to 238, and that drop says more about how few events are being counted than about the service. Services that fail rarely should be plotted by quarter rather than by month.

"MTBF was 90 hours last quarter and it's 260 this quarter."

Measured the same way, the gap between failures has widened nearly threefold.

"MTBF improved but MTTR is flat."

It breaks less often, and when it does the service is still down just as long.

"That one's severity 3, so it's out of the MTBF count."

It doesn't meet the agreed bar for an incident, so it isn't in the numerator.

At Pagedeck, Theo compiles this value each quarter and sets it beside the list of postmortems. Last quarter there were five incident writeups producing twelve improvement items, and in the quarter where eight of them got done the value went from 90 hours to 260. The number alone explains nothing; next to the list of finished items it explains itself.

Mia uses it for something else. It informs how much new work goes into the next quarter's plan. In a quarter where the gap slid back under 100 hours, she carves out stabilization time before committing to new features.

Reading the average as a countdown is one mistake. At 240 hours it's tempting to assume the next ten days are safe. Failures don't spread out evenly, though. A month where two incidents land on deploy day and the following three weeks stay quiet averages to the same 240 hours. Predict the next failure from this number, thin out the on-call rotation accordingly, and nobody is around on the day they cluster.

Reading a longer gap as a more stable service is the other. The number went up, so it looks like progress. But this value also rises when the bar for counting an incident is raised. In a quarter where severity 3 was dropped from the calculation, the value triples without anything being fixed. Set it as a target and the team drifts toward a stricter bar, so the number climbs while customers hit exactly as many failures as before. What's worth reading is the movement while the counting rule stays put.