Service Level Indicator

There was a time when "is the service okay right now" got a different answer from every seat. The person watching the servers says the process is alive and the CPU is at 40 percent, so everything is fine. The person reading support mail says saves are failing and the tickets are piling up. They were looking at different things, so both were right.

The trouble was that the numbers being measured described the system's own situation. CPU utilization, free memory, remaining disk are what you need to look after a server, and none of them connect directly to what a user goes through. Nobody experiences CPU. People experience whether pressing save saves, and how long they wait for it.

Theo had exactly that morning. Every dashboard was green and twelve tickets were waiting. Save requests coming from one region were timing out, and no server-side number showed it. After that the team moved what it measured over to the actions users take. A service level indicator (SLI) is the name for a number measured that way.

Divide good events by total events and multiply by 100. Three decisions produce the number.

1. Pick one action users actually perform. A save request, opening a document, logging in — something a customer notices immediately when it fails. Define it per request rather than per screen; requests are countable.
2. Decide what counts as good. Writing that threshold down explicitly is half the calculation. For a save request, good might mean the server returned a success response and that response arrived within two seconds. A response that arrived after nine seconds is not good.
3. Fix the period. The usual choice is a rolling 28 days, recalculated daily. Calendar months reset the number on the first, which erases last month's incident from the screen on a date with no engineering meaning. A rolling window keeps the same length of history in view whatever today happens to be.

If 12.4 million save requests arrive over 28 days and 12.38 million of them meet the threshold, the SLI is 99.84 percent. The raw material for it is metrics, the observability data that counts values of one kind and stacks them along a time axis.

Things like response time, which do not divide neatly into good and bad, get written as a percentile instead — the value with the slowest one percent set aside. Averages don't work here. A service averaging 0.3 seconds can still have one user in a hundred waiting nine seconds, and the average deletes that user. That user might be the largest account.

A good SLI is one that moves in the same direction as what users feel, not one that reads high. When the number drops, tickets should rise; when it recovers, tickets should fall. An indicator that stays flat while complaints climb is measuring the wrong thing.

Three conditions to check when choosing one.

1. It sits close to the user's experience. Save success rate is something users live through; CPU utilization is not. The latter helps find causes and does not measure quality.
2. The team can act on it. When the number drops, the thing to fix has to be inside your code or your configuration. Fold the state of the user's mobile network into the measurement and a bad number points at something you cannot change.
3. There are few of them. Two or three per service is about right. Start measuring twenty and some of them are red on any given day, at which point nobody reads any of them. Two lines get read every morning; twenty become a wall people scroll past.

Putting a target on the number is the next step, not part of this one. An SLI only measures; the line saying how much of it you commit to holding is the service level objective. The slack left under that line is the error budget: with a target of 99.9 percent, failing 0.1 percent of the time is allowed. None of those three exist without a measured value, which is why the indicator gets defined first.

"Let's make save success rate the SLI, with good defined as a success response inside two seconds."

Fixing for the whole team what gets measured and what counts as good.

"This one is hard to use as an SLI. When it goes bad there's nothing on our side to fix."

The value sits outside the team's control, so it cannot support a decision.

"We're at 99.84 percent over 28 days. Last week's incident is still inside the window."

The period rolls, so a recent incident is still reflected in the number.

Pagedeck keeps exactly two: save success rate and the response time for opening a document. Theo instrumented both so they can be broken out by region, which makes the state where the overall number holds while one region degrades visible on the screen. Opening the dashboard in the morning means reading two lines.

Mia uses those two lines in feature meetings as well. In a week where save success rate has fallen below target, new work gets pushed and someone is assigned to the area instead. The number does not make the decision, but the conversation about what comes first starts from those two lines.

The most common mistake is reading a high number as users being satisfied. At 99.9 percent it looks like nearly everyone is fine. But the number averages everything into one bucket, which can hide a group that fails constantly under one particular condition. If 0.1 percent of requests fail and that 0.1 percent is every request from one large account, the service is 100 percent broken for that customer. Leave the mistake in place and churn happens without ever appearing on a dashboard. Breaking the number out by account size or by region is what makes that group visible.

The second is treating it as a scorecard for the engineering team. A named number looks convenient to evaluate people with. But it exists to decide what to work on first. Used for evaluation, the team finds the cheapest way to raise it, and the cheapest way is editing the definition rather than the code. Move the threshold from two seconds to five, or drop the request type that fails most out of the count, and next month's number is higher. The value on the screen improves and the support queue grows.