Metrics

There was a long stretch where customers were the ones who told you something was wrong. Once a dozen support tickets piled up, an engineer would log into the server and open whatever logging had written down. Logging records what happened during a run, one line of text per event. The answer was in there somewhere, but finding it meant scrolling through a file with millions of lines with no idea what to look for.

The harder problem was having nothing to compare against. Suppose you do find three failed document saves in the file. Is three normal for a Tuesday afternoon, or is today unusual? The only way to know was to open yesterday's file and count the same lines by hand, which nobody does while an outage is running.

Metrics exist to take both of those problems away at once. Instead of writing down the story of each event, the system counts one kind of value and puts it on a timeline. Record the number of failed saves every minute and comparing two o'clock today against two o'clock last week becomes a single glance at a chart. The trade is deliberate: metrics give up on explaining what happened and why, so they can answer when it changed and by how much.

Every reading is stored as a name, a number and a timestamp, and later the readings that share a name are strung together in time order and read as a chart.

A single point has four parts.

1. Name — what is being counted. Saved documents might be document_save_total.
2. Value — the number at that moment.
3. Timestamp — when the reading was taken. Without it there is no timeline.
4. Labels — tags that split one name into separate lines. Tag each request as succeeded or failed and you can chart the failures on their own later.

There are three ways to count.

1. Counter — a value that only goes up. Total requests served never goes down. It works like the odometer in a car, which is why you read the difference from a minute ago rather than the number itself.
2. Gauge — a value that moves both ways. Memory in use or people currently connected; the current reading is the answer. Staying with the car, this one is the speedometer.
3. Histogram — a count of how many readings landed in each range. Used where the shape of the distribution matters more than any single reading, like response time.

Histograms exist because averages lie often enough to be dangerous. If 95 of 100 requests finish in 0.1 seconds and 5 take 8 seconds, the average is 0.5 seconds and the chart looks fine. But one user in twenty waited 8 seconds, and that user is the one who files a ticket. So response time gets read as percentiles instead. A 95th percentile (p95) of 8 seconds means that once you cut off the slowest 5 percent, the boundary sits at 8 seconds.

Thousands of readings per second are not all kept as they arrive. They are usually rolled into one point per minute, and older stretches get rolled up more coarsely still. Nobody goes back to second-by-second data from yesterday afternoon, but last quarter's trend still has to be readable.

Collected numbers do real work in four places.

The first is the dashboard. Request volume, failure rate and response time sit on one screen and someone skims it in the morning. The point is to learn the normal shape, so on a strange day the shape gives it away before anyone reads a number.

The second is alerting, the mechanism that contacts whoever is on duty when a condition is crossed. Writing that condition down requires something countable: "failure rate above 1 percent for five minutes" is not expressible without metrics. Nobody is watching a dashboard at three in the morning, so in practice this is the busier of the two.

The third is the minutes right after a deploy. Ship a new version, watch failure rate and response time for ten minutes, and roll back to the previous version if they climb. Being able to settle the rollback question with one chart is most of the value here.

The fourth is setting targets and judging them. A service level indicator (SLI) is the number that measures service quality, and a service level objective (SLO) is the line a team promises to hold it above. Both sit on top of metrics. A target of 99.9 percent save success only means something if successes and total attempts are already being counted, and whether the month stayed above the line is settled with the same numbers.

"Is this covered by a metric? If it isn't on a dashboard it may as well not be happening."

A request to add instrumentation before the feature ships. If it isn't being counted, nobody finds out when it breaks.

"Show me p95 instead of average response time."

Averages hide the slow minority, so look at the tail of the distribution instead.

"Don't put user IDs in labels. That will blow up the time series."

A warning that a tag with too many possible values multiplies the number of lines that have to be stored.

At Pagedeck the same scene repeats on every release day. Theo starts the deploy, leaves the failure rate chart open, and goes back to other work. If the line moves from its usual 0.2 percent up to 3 percent he rolls back immediately, and only then opens the logs to find out what broke. Hunting for the cause first just means customers keep hitting the failure the whole time.

The most common mistake is expecting metrics to explain an outage. When a chart bends at exactly 14:01 it feels like the answer is right there, which makes the assumption a natural one. But metrics answer when and how much. Which component was slow is a question for distributed tracing, and what blew up inside that component is a question for logging. A team with only metrics gets to 14:01 every time and then goes digging on the server by hand.

Believing that more data is always better is the other one. Measuring more sounds like knowing more, but the cost of metrics scales with the number of label combinations rather than the number of points. Attach a user ID label to response time and a service with 40,000 users now has 40,000 separate lines to store. Storage cost jumps by an order of magnitude and the charts get too slow to open. Labels should carry values you could list on one hand, and anything that needs to be viewed per user belongs in logging.