Percentile

The first thing anyone reaches for when reducing a pile of numbers to one is the average. But an average lets the fast values hide the slow ones. Ten response times made of nine at 0.2 seconds and one at 3 seconds average to 0.48. Nobody experienced 0.48, and the person who waited three seconds does not appear in that number at all.

Which leaves two people in a meeting looking at different things. Support keeps receiving complaints that the product is slow while the dashboard average sits comfortably in range. The person reading complaints is hearing from people who genuinely had a slow experience; the person reading the number is looking at a value those experiences were erased from. Neither is lying and the conversation goes nowhere.

A percentile orders the values and points at a position instead of flattening them. Saying what the value is at the 95 percent mark keeps the slow end of the experience inside the number, which is what makes the two sides of that meeting able to talk.

Sort the values from smallest to largest and read the one at the position you want. Three steps.

1. Order every value by size. Unlike an average there's no addition involved; there is sorting, which is why percentiles are more expensive to compute over large sets.
2. Choose the position. The 95th percentile is the point 95 percent of the way up, meaning 95 percent of observations came in below that value. More than one convention exists for interpolating between two neighboring values, and they disagree only at small counts.
3. Read the value sitting there. With 100 observations, it's the 95th one.

Take ten response times of 0.1, 0.1, 0.2, 0.2, 0.2, 0.3, 0.3, 0.4, 0.9 and 3.0 seconds. The 50th percentile is 0.25 seconds and the 90th is 0.9. In practice these get written p50, p95, p99.

Observation count is the thing to watch. Talk about a 99th percentile over fewer than 100 observations and you're pointing at the single largest value. If that one happened to be slow by accident, tomorrow's number is entirely different. So whether to go to p99 or stop at p95 gets decided by how many observations accumulate in a day.

Setting it beside the neighboring summary statistics shows its character. The mean, everything added and divided by the count, gets dragged by one very large value. Change that 3.0 to 10 seconds in the set above and the mean jumps from 0.48 to 1.18 while the 90th percentile stays at 0.9. That's why percentiles hold steady when an outlier, a value far from the rest, is mixed in — and why the upper percentiles are also the window for looking at those outliers.

It plays a different role from the measures of spread. Standard deviation, variance brought back to the original unit, summarizes how scattered everything is in one figure. A percentile names the value at one position.

The first decision isn't what value is good; it's which position to read. Response speed gets watched at 95 or 99, usage distributions at 50 and 90 together. A target with no position attached can't be judged.

The reason for going all the way to the top 1 percent is user count. On a product taking 200,000 requests a day, 1 percent is 2,000 of them, and a large share of those land on the same people repeatedly. The composition of a screen matters too: if one screen is drawn from ten requests, each request's slowest 1 percent appears roughly ten times as often at the screen level.

Targets get written by attaching the position to the value. "p95 under 1.5 seconds" can be judged on the day. "Average under 1.5 seconds" can pass while the number of people experiencing three seconds grows.

Not mixing populations is part of the standard. A p95 combining regions, devices, and plans points at none of them. Split phones from desktops and one side often reads 1.1 seconds against the other's 4.2. That's how a combined 2.1 seconds clears a target while half of phone users are living with four.

Finally, state the time window. A p99 computed over five minutes and one computed over a full day are different numbers, and the shorter window catches incidents sooner.

"The average is 0.4 seconds and p95 is 3.1."

Most of it is fast and a slice of it is badly slow.

"This target has no position on it. Is it p95 or p99?"

Rewrite the target in a form somebody can judge.

"There are only 80 requests, so p99 means nothing here."

Too few observations, so that position rests on a single value.

At Pagedeck, when complaints about documents opening slowly kept arriving, what Theo looked at was p95 rather than the average. The average was 0.4 seconds and p95 was 3.1. Everyone who had written in was inside that three-second band.

Splitting the data showed it was only accounts with many tables inside their documents. Mia had the dashboard's average removed afterwards and p50 and p95 placed side by side, because the gap widening between the two is itself the signal. A p50 holding steady while p95 climbs means some subset of accounts is diverging, not that the product got slower for everyone.

The most common mistake is averaging percentiles. With four servers and a p99 from each, it's tempting to average the four and call it the overall p99. But this value was read from a position in a sorted list, so it can't be added or divided. Four servers each at a p99 of 2 seconds can combine into an overall p99 of 3. Getting the real number means merging the raw observations into one list and sorting again. Miss this and the reported figure runs optimistic while the complaints keep arriving regardless of it.

The second is expecting a lower p99 to make everything faster. Fixing the slowest slice sounds like it should lift the rest. But what makes the top 1 percent slow is usually not what makes the average what it is — a first request doing setup work, an account with unusually much data, a brief burst of traffic. Fix the setup work and p99 can fall from 3.1 seconds to 1.8 while p50 doesn't move off 0.2. Not knowing those are two separate jobs means fixing one, finding the other unchanged, and going back to look for a cause that was never shared.