Alerting

For a long stretch of software history, the people using a service found out it was broken before the people who built it. You can put as many graphs on a wall as you like; if nobody is looking at them, they say nothing. Document saves can start failing at two in the morning and nobody in the company knows until someone walks in and opens a screen.

So incidents usually arrived from the outside. Three or four support tickets would pile up, the support agent would ask engineering, and engineering would finally go digging through the records. The seven hours of failures in between were a signal that would have been visible in the first five minutes to anyone sitting in front of the screen.

The problem was never a shortage of signal. Metrics were stacking request counts and failure rates second by second, and logging was writing a line for every failed request. Both were accurate, and both waited for a human to come and look. Recording what happened and summoning a person are different jobs, and nothing among the observability signals handled the second one. Alerting fills that gap by writing the condition into the system ahead of time and letting the system make the first call.

What alerting closes is the gap between the moment a signal appears and the moment a person notices it, and the length of that gap is the size of the damage. Notice seven hours late and you have seven hours of failures; notice five minutes late and you have five minutes of them.

Closing the gap means writing down, in advance, the judgment a person used to make on the spot. A single rule usually has four parts.

1. What to watch — pick the value. The closer it sits to what a user actually experiences, the better. Memory usage at 90 percent on one server can be true the whole time nobody notices anything wrong.
2. Where trouble starts — decide which line counts as a problem. If the team has an SLO, a service quality target it has committed to, borrow that line directly. A 99.9 percent save success target makes a 0.1 percent failure rate the threshold.
3. How long it has to last — one spike is not worth a phone call. Failure rates jump for a few seconds right after a deploy all the time, so conditions carry a duration: a failure rate above 1 percent sustained for five minutes.
4. Who gets it and how — who is contacted, and who is next if nobody answers. The person on duty at that hour is on-call, and the procedure for passing an unanswered page upward is escalation.

Drop the third part and the phone rings on every deploy. Drop the fourth and the alert fires into an empty room.

The tradeoff never changes. Set the line low and you get called for things that aren't problems; set it high and you miss real ones. It works like a smoke detector: wire it to react to a wisp and it goes off every time someone cooks, until the batteries come out. Alerts get disabled the same way, except the habit of ignoring them replaces the act of removing a battery.

The oldest option is to have people watch. Put a dashboard of key metrics on a wall and react to whatever stands out. During office hours this works well, and a person reads the whole screen rather than one number, which catches oddities nobody could have written a rule for. What it does not cover is nights and weekends.

A scheduled review is another substitute — a summary of the last 24 hours in your inbox every morning. It costs almost nothing to set up and it is actually better at slow decay. A response time that drifts from 0.4 to 1.1 seconds over a month never crosses a line on any single day, so no threshold rule catches it. What it cannot do is tell you about something happening right now.

Anomaly detection replaces the human-chosen threshold with a learned one. It models the normal shape of a value and reports departures from it. No fixed line can encode the idea that traffic halving on a Friday evening is fine while traffic halving on a Tuesday afternoon is not. The weakness is that the reason is vague: "failure rate above 1 percent" lands with its meaning attached, while "this looks unusual" makes the reader start from scratch.

Error tracking, which groups exceptions of the same kind together, also carries a paging feature. It handles conditions that are hard to express as a magnitude, such as a new kind of exception appearing for the first time.

"Did nothing fire for this? The failure rate has been at 8 percent for half an hour."

Either no rule covers this value or the threshold sits too high.

"That one is noise. It fired fourteen times this week and recovered on its own every time."

A proposal to delete the rule or attach a duration condition to it.

"Let's decide whether this is worth waking someone for. Otherwise it goes on the morning list."

Separating what needs a hand now from what can wait until tomorrow.

At Pagedeck, Theo goes through the week's pages at the end of every on-call rotation. If four of them reached a phone and only one led to anyone touching anything, the other three get treated as badly written rules — the condition changes or they stop firing overnight. When a problem reaches incident management through a support ticket first, he checks why no rule caught it and writes a new one.

More rules feel safer. Tightening the net sounds like the right instinct. In practice it runs the other way. Forty pages a day and the person receiving them stops opening any of them, and once that habit forms the one real page goes by with the rest. Left alone, an outage ends with a perfect record of a message that nobody read. Teams that run this well don't count how many rules they have; they count what fraction of pages led to someone doing something.

The other common mistake is treating a page as the end of the response. Sent is not received, and received is not fixed. Alerting only announces the start, and what follows belongs to the incident that opens after it. Who owns it, where the current state gets written down, and what gets kept afterward are agreements that live outside the alert.