Incident Management

There was a time when nothing anywhere said what to do when the service stopped. The first party to notice was usually the customer, not the company. Five support tickets saying "saving doesn't work" would pile up before anyone felt something was off, and whether that person happened to be at a desk decided how long the outage ran.

So outages always played out the same way. Somebody posted "is anyone else unable to connect right now?" in the engineering chat, and three people who read it logged into the servers and each poked at whatever they suspected. One person reverted a setting that another quietly changed back, and outside the building customers sat refreshing a page with no word from anyone. When the screens came back, that was the end of it. Nobody wrote down what had gone wrong, and two months later, when the same cause took the service down again, the team wandered through it as if it were new.

The wandering happened in two separate places, and that split matters. One was the absence of any agreement about who does what while the fire burns. The other was the absence of anything left behind once it goes out. Incident management covers each with a different mechanism. The first gets covered by assigning roles and an order of operations in advance, so nothing depends on who reacts fastest. The second gets covered by treating recovery as the middle of the job rather than the end, with the written record as the finish line. What the term names, then, is less a repair skill than an agreement about how repairs are run.

Within engineering, the work of watching a service while it runs splits three ways. Observability is the job of planting logs and metrics ahead of time so you can tell from outside what is happening inside the system. Incident management takes over when something actually stops, deciding who moves first and what gets restored in what order. DORA metrics measure how often and how safely the organization ships, across four values. The three interlock. With nothing planted, incident management never receives its starting signal, and support tickets become the detection mechanism by default.

Pagedeck lived through exactly this. One Saturday afternoon document saving failed for 18 minutes, and Theo only learned about it because Sofia took a call from a customer contact. No alert had fired. The service came back that evening, but the question Grace asked pointed somewhere else: if the same thing happens at three in the morning on a Saturday, how many minutes pass before anyone knows? Nobody in the room could answer. Not because the team lacked skill, but because nobody had ever decided the answer in advance. Theo had been the one looking at servers that Saturday only because he happened to pick up the phone.

The terms grouped under this one fall into four branches: the concept that defines what counts as an incident, the structure that says who moves and when, the aftermath that says what happens once the fire is out, and the metrics that measure whether any of it is working. The order of the four is also the order a single incident travels through.

The concept branch draws the line between what counts and what doesn't. An incident is an event that actually affected the service. It gets its own name because an error is not automatically an incident. Server logs collect thousands of error lines a day, and only the ones that stopped a user from doing something they were trying to do get counted. Without a written test for that, one person calls a situation an incident and another doesn't, and counting anything becomes impossible later on. Most teams grade incidents by blast radius and severity — every user unable to save is a level one, text rendering badly in a single browser is a level three — and the grade decides who gets called and how soon. Skipping grades goes wrong in both directions. Treat every anomaly as equally heavy and people burn out, so response slows down exactly when a real outage lands; treat every anomaly as light and the real outage gets noticed late.

The structure branch places people. Two terms answer two different questions.

1. On-call — the person on duty to receive outage calls during a defined window. Rotations usually run a week at a time, and the assignment comes with a promise to respond within a set number of minutes. Naming a single person matters because an alert sent to everyone belongs to no one. Ten people who get the same page all assume one of the other nine is already looking. Most rotations also name a secondary, since an incident will not politely wait for the primary to get off the subway.
2. Escalation — the procedure for handing an incident upward or sideways when the person on duty can't close it. The trigger is set in advance as a clock: if the cause hasn't been narrowed within 30 minutes, call the senior engineer, no discussion. Fixing the condition as a number keeps nobody standing alone at three in the morning weighing whether it's acceptable to wake someone. Every minute spent weighing that is a minute the outage runs.

The aftermath branch handles what comes after the fire is out. Three terms cover the record, the analysis, and the attitude.

1. Postmortem — the written account of a single incident. It lays out when the problem started, when someone noticed, and what actions brought the service back, in order; it closes with what will be fixed, by whom, and by when. Teams usually finish it within a few days, while the details are still sharp. The intended reader is a colleague who wasn't there, and that shapes the format: it has to name the log that was checked and the button that was pressed, so that someone meeting a similar symptom six months later finds their answer in it.
2. Root cause analysis — the practice of following a symptom down to what actually produced it. It doesn't stop at "the disk filled up." It asks why it filled, and why nobody saw it coming. The thing genuinely worth fixing almost always sits at the last answer. Clearing the disk is recovery; making an alert fire when free space drops is prevention.
3. Blameless postmortem — the stance of fixing structures instead of naming people. It looks like an optional nicety, and without it the other two collapse. Once a team watches one person get called out for a mistake, the next person writes a vaguer account of what they did, and a vague account yields no cause.

The metrics branch turns all of it back into numbers. Mean time to recovery (MTTR) is the average time from the start of an outage to the moment service returns. Watching it fall from four hours to 40 minutes is read as the duty roster and the procedure settling into place. Mean time between failures (MTBF) is the average gap between one incident and the next. The two are always read together. Track only MTTR and a team gets very good at putting out fires that keep starting; track only MTBF and a single outage can run all day without the number looking bad. They also respond to different levers. The first drops quickly through moves like a rollback to the previous version, while the second only stretches when the fixes named in postmortems actually get done.

The four branches aren't a sequence of stages so much as a loop. Without the concept branch deciding what to count, the metrics report whatever they like; without the structure branch, the gap between when an incident begins and when someone notices grows, and MTTR breaks first; when the aftermath branch becomes a formality, the same cause returns every quarter. Run the loop properly once and its output feeds the next turn: one improvement from this incident shortens detection time for the next, and faster detection cuts recovery time again.

"This is a level one and Theo is on call this week. If we haven't narrowed it in 30 minutes I'm escalating."

The grade, the duty assignment, and the handoff point are all settled already, so there is nothing to negotiate in the moment.

"Let's find the cause later and roll back now. Recovery comes first."

A reminder that while an incident is live, getting back to normal outranks understanding what happened.

"This review isn't about deciding who made the mistake. It's about the path that let that deploy through without anything stopping it."

A statement that the subject is the structure that left room for the error, not the person who made it.

Pagedeck built its procedure over the month after that Saturday. Theo put four engineers into a weekly rotation, and Grace wrote the severity grades onto a single page. At level one, the person on call responds within five minutes, Mia posts a notice inside the product, and Sofia calls the larger accounts first. At first there was some grumbling that writing this down was heavy for a 45-person company. The fact that the whole thing fit on one page is what settled the argument.

The next outage arrived two months later on a Wednesday night. Theo answered the page in seven minutes, never found the cause, and brought the service back in 22 minutes by reverting the most recent deploy. The cause turned up the following afternoon. What Mia raised in the review wasn't the 22 minutes. It was that the 18 minutes nobody knew about in the Saturday incident had become seven, and that the difference came from a duty roster rather than from better people.

The most common mistake is reading incident management as the work of fixing things quickly. The urgent half is the visible half, so the assumption is natural. In practice about half the work happens after the fire is out. Skip the aftermath branch entirely and a team gets steadily better at extinguishing the same fire. That's how you end up with recovery time down from two hours to 25 minutes over a year while incident counts climb quarter over quarter from the same cause.

Setting a target of zero incidents is the other one. Fewer incidents are better, so it feels like the right line to write down. But that number reaches zero along two different paths: the service genuinely stabilizes, or incidents stop being called incidents. The second path is easier. Once the count is tied to evaluation, people leave borderline cases off the severity list, defer risky changes and then ship them all at once, and let a 20-minute outage pass quietly. Recorded incidents drop while the incidents customers experience stay exactly where they were, and an incident nobody recorded teaches nobody anything. So instead of targeting the count itself, teams watch values that are harder to hide — time to recovery, or how often the same cause comes back.