Statistics

Producing a number stopped being the hard part a long time ago. A dashboard shows average response time, an experiment tool announces which variant won. What still happens, regularly, is that the same number gets read two opposite ways by the people sitting in front of it.

Pagedeck ran into this directly. Mia brought an average response time of 1.2 seconds; Theo said complaints about slowness arrived every week. Both were right. Half of all requests finished within 0.3 seconds while the slowest 5 percent took more than 6, and a single average had flattened those two groups into a middle that nobody experienced. The hour spent arguing over who was correct could have been five minutes if anyone had known which number ends that argument.

The other recurring fight is whether a difference is a difference. If the changed signup screen converts at 12 percent and the untouched one at 9 percent, the change looks like a winner. With only 200 people on each side, though, those 3 percentage points are not far from flipping a coin 200 times twice and getting 24 heads once and 18 the other time. Run the same test next week and the order can reverse.

Neither situation was caused by missing numbers. The numbers were already on screen. What nobody had checked was what got thrown away to produce them and how much they wobble from one run to the next. Statistics is the set of ideas used for that check. It covers how many values get compressed into one, what disappears in the compression, and how wrong you can be when you look at part of a group and speak about the whole. Tools make the number; these ideas decide how much weight it can carry. None of it requires deriving formulas by hand, since the software computes everything mentioned on this page. What it requires is knowing which of those outputs answers the question in front of you.

The terms under statistics fall into five groups: ways to compress many values into one, ways to describe how spread out the values are, ways to speak about a whole group from a part of it, ways to weigh whether an observed difference is chance, and ways to write the relationship between values as a formula. The first two only describe the data in hand. The last three attach whenever you try to say something about data you don't have.

Central values compress many numbers into one, and there are three.

1. Mean — add every value and divide by the count. Five response times of 0.3, 0.4, 0.3, 6.2 and 0.3 seconds sum to 7.5, so the mean is 1.5 seconds. Every value contributes something, which is both the strength and the weakness: not one of those five requests took anything close to 1.5 seconds.
2. Median — line the values up by size and take the middle one. The same five sorted are 0.3, 0.3, 0.3, 0.4, 6.2, so the median is 0.3 seconds. It doesn't move when one far-off value joins the set, which makes it closer to reality than the mean for anything with a long tail, such as salaries or response times.
3. Percentile — where a value sits in the whole range. A 95th percentile of 6 seconds means 95 out of every 100 requests finished within 6 seconds and the remaining 5 took longer. The median is itself a percentile, the 50th. Service speed gets read this way rather than as an average because the average hides the people having the worst time.

Spread describes how scattered the values are, and there are three. This is where central values run out: two groups can share a mean while one clusters tightly and the other splits toward both ends.

1. Variance — the distance of each value from the mean, squared, then averaged. The squaring keeps values above and below the mean from cancelling each other out. The cost is that the unit gets squared too, so a variance calculated on seconds comes out in squared seconds and can't be read directly.
2. Standard deviation — the square root of the variance, which puts the number back in the original unit. A task averaging 40 minutes with a standard deviation of 5 minutes mostly lands between 35 and 45. The same average with a standard deviation of 30 minutes sometimes finishes in 10 and sometimes runs past an hour. Only the first one is usable for scheduling.
3. Outlier — a value sitting far from the rest, like the 6.2-second request above. Sometimes it's an error to remove and sometimes it's the most important signal in the set, so the question of why it appeared comes before the question of whether to drop it. A payment 300 times the usual size is a typo or a new enterprise contract, and the next step is completely different in each case. If one does get removed, the removal belongs in the write-up next to the result, because a number with the inconvenient rows quietly deleted is a different number.

Estimation speaks about a whole group from a part of it, and there are two.

1. Sample — the part drawn to stand in for the whole. You can't ask all 120,000 users, so you ask 800 and treat their answers as everyone's. How they were drawn matters more than how many there are. A survey shown inside the app is answered by people who open the app often, so collecting 8,000 instead of 800 still never reaches anyone who left.
2. Confidence interval — the range the true value is taken to sit in, given what the sample showed. If 46 percent of 800 respondents say they're satisfied, the result gets written as somewhere between 42 and 50 percent. Larger samples narrow the range. Carrying the range alongside the number answers a question immediately: last month's 44 percent and this month's 46 percent overlap, so nothing has gone up yet.

Testing weighs whether an observed difference is chance, and there are three.

1. p-value — the probability of seeing a difference this large purely by chance when there is really no difference at all. A p-value of 0.03 means results like this one turn up three times in a hundred even in a world where the two options are identical.
2. Statistical significance — the judgment that the probability fell below a line set in advance, so chance is a poor explanation. The line is usually 0.05, and it gets fixed before the experiment starts. Moving it after seeing the result turns a judgment into an explanation written to fit.
3. Statistical power — the probability of catching a real difference when one exists. Small samples push it down, and a "no difference" result from an underpowered test doesn't mean there was no difference, only that the test couldn't have seen one. This is why the number of participants gets calculated before the experiment runs. A test with 200 people per side chasing 3 percentage points usually fails here.

Models write relationships between values as formulas, and there is one entry: regression. It estimates how several variables move together with an outcome and returns something like how many more documents get created for each additional person on a team. The warning that follows it everywhere is that a relationship found this way is not a cause. Even if teams that create more documents stay longer, whether pushing teams to create documents makes them stay is a question only a split test answers.

All five groups get used at once to read a single experiment, which applies a change to part of an audience to see its effect. Power decides how many people to gather, central values pick the number that summarizes the result, spread and confidence intervals say how much that number wobbles, and testing rules on whether it counts as a difference. Drop any one of them and the rest get misread.

"Show me the 95th percentile instead of the average response time. The average buries the slow end."

A request to look at the worst experiences rather than a number summarizing everyone.

"With 200 per side, a 3 percentage point gap doesn't tell us much. The confidence intervals overlap."

The observed difference is small enough that chance produces it, so hold off on the conclusion.

"This test was underpowered, so a null result doesn't mean there's no difference."

The sample was too small for the design to catch anything short of a huge gap.

The week Pagedeck's growth team tested new copy on the signup screen, this conversation happened almost line for line. Ravi's table showed 12.1 percent for the changed screen and 9.4 percent for the original, with 210 people on each side. Mia's first question was how wide the confidence intervals were. Calculated out, one ran from 8 to 17 percent and the other from 6 to 14 percent, overlapping heavily.

They left the test running for three more weeks rather than shutting it down. The real problem was that nobody had worked out the sample size beforehand; redone properly, separating a gap that size needed about 1,500 people per side. While Ravi had the data open, the two also split it by acquisition channel, and visitors arriving from search converted so differently from visitors arriving through invitations that the combined average described neither group.

The most common belief is that statistics belongs to people who studied it and that everyone else reads whatever the tool prints. Experiment tools do mark a winner and dashboards do compute averages automatically, so the belief has support. But a tool produces numbers and stops there; it has no idea which decision the number is about to be used for. Whether response time should be read as an average or a 95th percentile, when to stop an experiment, whether overlapping ranges count as a gap — only someone who knows the product settles those. Hand the judgment to the tool and you ship on the day the winner badge appears, then can't explain two months later why the metric looks the same. What the work needs is not the ability to calculate but the habit of asking what a number discarded on its way to being one number.

The second error is reading "significant" as "large" or "important," which follows from how the word is used everywhere else. In statistics it carries exactly one meaning: the observed difference is hard to get from chance alone. It says nothing about size. An experiment splitting 100,000 users per side can return 4.00 percent against 4.05 percent as a significant result, and rebuilding a screen for those 0.05 percentage points may still be a waste. The mirror image travels with it — reading "not significant" as confirmation that nothing differs. Small samples leave sizable gaps looking insignificant, so the only conclusion available in that case is that you still don't know. Results are read with three questions side by side: is it significant, how large is the gap, and is a gap that size worth anything to the business. Answer all three and the result survives the next meeting; answer only the first and someone reopens the decision a month later.