On-call

Services don't break only during office hours. Payments jam at three in the morning, login stops working on a Sunday, and for hours nobody inside the company knows it happened.

The more common failure was subtler than that. Send the alert to everybody and nobody treats it as theirs. An alarm lands in a channel with ten engineers in it, ten people assume one of the other nine is looking, and nothing happens until somebody asks half an hour later whether anyone is on this. The opposite pattern is just as bad: when the same one or two conscientious people answer every time, they burn out within six months, and after that the channel is back to nobody looking.

On-call is the agreement that closes both gaps at once. It fixes exactly one person as the recipient at any given moment, and it spreads that duty across the whole team by rotation. The mechanism that matters is deciding who answers before the outage rather than during it.

From the moment a page fires to the moment the write-up lands, the sequence has six steps. They're written down so that somebody woken at 3 a.m. has as few judgment calls to make as possible.

1. The page arrives. The machinery that contacts a person when a condition is crossed is alerting, and it reaches the on-call engineer's phone as a call or a vibration. A chat message won't wake a sleeping person, so the channel has to make noise.
2. The engineer acknowledges. Teams commit to responding within a few minutes, and if that acknowledgment never comes the page moves to the next person automatically. Outages don't avoid the nights when the on-call engineer is underground on a train.
3. Impact gets assessed. An event that actually stopped users from doing what they came to do is an incident, and the first call is whether this qualifies. Everyone losing the ability to save and one browser rendering text badly get handled differently even on the same night. Severity is the scale for that judgment, and the grade sets the pace of everything after it.
4. The service gets restored. Putting the service back matters more than knowing why it broke. If a deploy that just went out is the suspect, roll back to the previous version and find the cause in the morning. Spending two hours at 3 a.m. tracing root causes is the most common mistake in this whole sequence.
5. Stuck means handing off. If the problem isn't resolved within an agreed window, the next person gets called. That procedure is escalation.
6. The record gets written. Once the service is back, what happened goes into a postmortem. What the on-call engineer looked at and clicked during the night has to be in there, or the next person meets the same symptom with no help.

Two people usually hold the duty at any moment. Primary takes the page first, and if primary doesn't acknowledge within the agreed window it rolls to secondary automatically. Secondary isn't a nice extra. Point everything at one person and the entire alerting chain stops on the night that person's phone is dead.

Two more roles appear only in larger incidents. One is whoever knows the affected system best, pulled in when step 4 isn't enough on its own. The other is someone running the incident itself. That role exists because at a certain size the fixing and the telling have to happen in parallel. If the person with their hands in the system is also posting the status page update and briefing leadership, both jobs run late.

A shift is usually a week. Daily handoffs break up incidents that are still in progress, and a month leaves one person tense for a month straight. At the weekly handoff, the outgoing engineer passes along what's still open and which of last week's alerts the team decided to ignore.

How the clock gets divided varies. A company with people in several countries can give each group its own daylight hours so nobody wakes up at all. A company in one place has to split nights and weekends, and when it does it writes a matching rule that someone paged overnight comes in late or takes the day. Without that rule, an on-call week is simply a week with more working hours in it.

"Who's on call this week?"

Asking which single person is currently holding the pager.

"I got woken twice last night and both pages were ignorable."

Reporting that the alert thresholds are loose enough to call people for things that don't need a human.

"Roll it back now and we'll look at the cause tomorrow."

Restoring service comes first, so don't start a root cause hunt in the middle of the night.

Pagedeck has eight engineers, and six of them including Theo take the rotation a week at a time. The week ends with a fifteen-minute handoff on Monday morning where the team walks down every alert that fired, finds the ones that didn't need a person, and tightens those conditions. Overnight pages dropped by more than half after that meeting started.

When Theo is on call, Mia doesn't schedule feature work for him that week. The duty isn't sitting and waiting; it's a week where a single incident can erase an entire day, so anything planned into it tends to slip.

The most frequent misreading is that on-call means sitting at a desk for twenty-four hours. Agreeing to take calls sounds like agreeing to stay put, so the confusion is understandable. The actual commitment is a response time, not a location. Answer within the agreed number of minutes and it makes no difference whether you're at the grocery store or in a cinema; the real constraint is staying somewhere with signal and a laptop within reach. Teams that don't draw that line end up with nobody willing to take the rotation, which collapses it back onto the same few people.

The other misreading is that the on-call engineer is supposed to fix everything personally. The job runs as far as assessing the situation, starting recovery, and calling whoever else is needed. That's why someone who has never read another team's code can still hold the pager, and it's exactly what step 5 exists for. Leave this one uncorrected and the on-call engineer hesitates to wake anybody for a problem they can't solve, and recovery loses every minute of that hesitation.