Incident

Server logs collect thousands of error lines a day. Only a handful of them are worth waking somebody up for. On teams with no word separating the two, every anomaly carried the same weight, and that failed in both directions. When everyone piles onto every alert, people burn out and respond slowly to the one outage that matters. When every alert gets waved off, the real outage goes unnoticed for an hour.

In practice it showed up as an argument. Someone would post "is this an outage or not?" in the team channel and get a different answer from every person who replied. One treated any single inconvenienced user as an outage. Another held out for the checkout page going down. Different thresholds produced different responses: one night ten people stayed up over a broken font color, and another night half the users couldn't save a document while nobody was paged at all. At the end of the quarter, asking how many outages there had been produced a different count from every person who tried to answer.

Incident exists to move that judgment out of the channel and into a rule written down in advance. Only events that actually affected the service get the name, and the moment the name is attached, a defined procedure starts. The distinction the word carries is simple: an error is not automatically an incident. A retry that succeeded on the second attempt produced an error line and no incident, because the person on the other end never noticed anything.

An incident runs through four stages — declare, grade, respond, close.

1. Declare. It begins when somebody calls the situation an incident. The authority to declare is deliberately spread wide. Being wrong costs one unnecessary meeting; staying quiet about something that turns out to be real costs hours of downtime.
2. Grade. A severity level sets the weight. Nobody can save a document, that's a level 1; text renders badly in one browser, that's a level 3. The scale measures blast radius and disruption, not how hard the fix is. A one-line bug that keeps everyone from logging in is still a level 1, and a fix that takes three days but affects one customer's export is still a level 3.
3. Respond. The grade decides who gets pulled in. A level 3 waits for the on-call engineer during business hours. A level 1 pages that engineer immediately, and if it isn't contained within a set number of minutes it climbs the escalation path. During response one person runs the incident while the others split up the investigation, so nobody has to ask who is deciding.
4. Close. The incident closes when users can work normally again. Three timestamps get recorded — when it started, when it was detected, when service came back — because metrics like mean time to recovery (MTTR) can't be computed without them.

The word travels through three places. The first is the minute after an alert fires. When the on-call engineer looks at a dashboard and says "I'm raising this as an incident," a dedicated channel opens and everything else on the calendar moves back.

The second is the write-up after the fire is out. A closed incident gets a postmortem: a timeline of when it started, when someone noticed, and what brought it back, ending with a list of fixes and the name and date attached to each one. Inside that document, root cause analysis (RCA) pushes past the symptom. It doesn't stop at "the disk filled up" — it asks why the disk filled and why nobody knew before it did. None of this survives without blameless culture. Once people watch a colleague get grilled for a mistake, the next person writes a vaguer account of what they clicked, and a vague account yields no cause. The document is written for a colleague who wasn't in the room, which is why it records the exact log lines and the exact screens involved.

The third is the quarterly count. Teams tally incidents by severity and check the uptime promised in the customer service level agreement (SLA). This is where the declaration rule pays for itself. Without it the count doesn't mean anything, because each person counts something different — one report says six for the quarter and another says twenty-three, and neither number can support a decision. The same rule is what makes a trend readable a year later, when the team that wrote the first count has half turned over.

"Should we raise this as an incident, or just file a ticket?"

Asking whether to start the response procedure or add it to the normal work queue.

"Calling it a level 2, channel is open. I'll run it."

The weight is set and the person coordinating has identified themselves.

"Service is back but I haven't closed it. We still don't know the cause."

Users can work again, but the record stays open because it could happen again.

At Pagedeck, autosave failed for twelve minutes one afternoon. Theo got the alert and declared an incident within five minutes, then raised it to level 1 once he saw what share of users were affected. While he worked, Mia drafted the notice that would go out to customers. After recovery Theo owned the write-up and Mia collected the list of customers who had written in.

All the severity level did was set priority. The moment level 1 was attached, the deploy scheduled for that afternoon stopped and two meetings were pushed. Without a shared name and a shared scale, that call has to be negotiated from scratch every time.

The most common misreading is that an open incident means somebody made a mistake. Something broke, so surely someone is at fault — a natural assumption. But the name describes the impact users felt, not the cause. A payment provider going down is an incident. More traffic than expected is an incident. When that distinction blurs, declaring starts to feel like an accusation, and every minute of hesitation is a minute added to recovery.

Setting a target of zero incidents fails the same way. A falling count looks like a hardening service, but the easiest way to lower it is to stop declaring. Teams handed that target drift toward filing the ambiguous cases as tickets, which leaves no record and therefore no improvement. Watch the distribution across severity levels and the time to recovery instead of the raw count. If level 1 incidents drop from one a quarter to none while level 3 incidents rise from twenty to thirty, the service didn't get worse — the team got better at catching small problems early.