Site Reliability Engineer

When development and operations sat apart, their goals pushed against each other. Development is judged on shipping features; operations is judged on nothing happening. A deployment is a change and a change is the raw material of an incident, so operations moved toward slowing deployments while development pushed to speed them up. Both incentives are rational on their own, which is why the standoff is stable.

The same shape appeared at Pagedeck. After a few incidents, deployments dropped to once a week, which made each release carry more change. Larger changes made causes harder to find, which made deploying scarier still. The loop tightens on itself.

What incidents left behind was the other problem. Theo recovered one alone at 3 a.m., the retrospective ended with an agreement to be careful, and something similar happened the following month. Care disappears when people change or forget.

What this role inverts is treating reliability as a number rather than an attitude. Set a target for how reliable the service will be, then calculate how much failure is allowed while still hitting that target. As long as that allowance holds, deployments do not get blocked. Making both sides read the same table instead of arguing is where the role starts.

There is no fixed day, but the time goes to six places.

1. Reading the numbers. Look at the service level indicator (SLI), the measured quality of the service, and at how much room is left against the service level objective (SLO), the target that was agreed. How much of the error budget remains this month sets the day's judgment: with room left, features ship; spent, people get assigned to reliability work. The budget is what turns reliability from an opinion into an amount.
2. Taking the rota. When the on-call shift comes around, alerts arrive and get picked up inside an agreed time. What matters on that shift is fast recovery; establishing the cause waits until afterward. The rota is written down in advance, including who is second if the first person does not answer.
3. Recording the incident. Once recovery is done, a postmortem gets written: what happened in what order, why it was not noticed sooner, and what change prevents a repeat. Not blaming anyone is a rule, because once blame starts, the facts stop getting written down. The document is read by people who were not there, so the timeline matters more than the conclusions.
4. Moving manual work into code. Opening account permissions by hand, adding servers by hand. A ceiling of about half the working time is set for that kind of work, and past it, time goes to automation instead.
5. Making the system visible. Raising observability — how much can be known from outside about what a system is doing. Without logs, metrics, and traces in place, an incident tells you nothing beyond the fact that a screen will not load. The work is cheap before an incident and impossible during one.
6. Laying in deployment safety. Progressive delivery, which widens the audience gradually instead of opening to everyone at once, and a prepared way back. On a deployment with no way back, finding the problem leaves nothing to do about it, and rehearsing that way back is part of laying it in.

The most frequent partners are the people building servers. With Theo the meeting happens while a feature is being designed, deciding what happens when this feature fails and how long to wait when an external service does not answer. Arriving after it is built turns the same conversation into a structural change, which is why reliability questions get asked at design time.

The other four have different purposes.

1. The DevOps engineer, who automates deployment and operations, overlaps most. In a small organization one person holds both.
2. Mia, for trading features against reliability. Postponing a feature in a month where the error budget is gone gets decided here, and because the decision rests on a number it does not turn into a contest of preferences. The same budget gives her a reason to say yes in months where it is untouched.
3. Grace, when writing the service level agreement (SLA) promised to customers in a contract. The contractual figure has to be looser than the internal target; a contract where that relationship is inverted is a promise that cannot be kept.
4. Sofia, for what to tell customers during an incident and when. Notification arriving after recovery reads to customers as us not having known.

"We've spent this month's error budget. Let's push the feature release to next week."

Using a number as the basis for deployment pace.

"Recover first, look at the cause in the postmortem."

During an incident, recovery outranks diagnosis.

"This isn't worth an on-call page. It can wait until morning."

Keeping the bar for waking someone up narrow.

At Pagedeck, with 45 people, one person holds this role. The highest-value thing that person did was cleaning up alerts. Thresholds had been set generously, so night pages arrived ten times a week and one of them needed action. Someone who has been woken nine times for nothing ignores the eleventh page too.

Now only situations where users genuinely cannot work wake anyone, and the rest collect into a morning list. Recovery times got shorter after the alert count came down, because the pages that did arrive got read.

Reading the role as the person who puts out fires is the most common mistake. During an incident this seat does move first, so it looks that way. But putting out the fire is the outcome; the work is changing the structure so the same fire does not start again. Assign only the firefighting and the person stays permanently busy while the incident count holds steady. Busy is not the same as effective here. That is why there is a rule capping the time spent on repetitive work.

The second is that the goal is 100 percent reliability. Not going down is a good thing, so it sounds natural. But the cost of the last few decimal places climbs steeply. Taking uptime from 99.9 to 99.99 means servers in several regions and more people on rota, and that money and time come out of features. Worse, a target of 100 makes blocking every deployment the correct choice, and the product stops. So the target is set below 100 and the difference is treated as an allowance to spend. Choosing that target is a business decision the role makes explicit rather than a technical one.