Statistical Significance

Comparing two numbers looks easy. If the screen with the new button color converts at 5.2 percent and the old one at 4.8 percent, the new one appears better.

But show the identical screen to two groups and measure, and the numbers still differ. Getting 4.8 percent and 5.1 percent from two groups where nothing was changed is routine, because people's circumstances differ day to day and which particular people happen to land in each group differs every time.

So teams comparing numbers without a standard kept repeating the same loop. See a difference, ship the new screen. Next month conversion is back at 4.9 percent. Several other things changed in between, so the cause can't be traced. Look back after six months and three of twenty experiments left any effect behind, and nobody can say which three were real.

Statistical significance exists to draw a line under that judgment: decide in advance that if a difference this large would arise from chance only rarely enough, you'll stop calling it chance.

The calculation runs backward. Instead of proving a difference exists, you compute the probability of seeing this result assuming no difference exists.

Four steps.

1. Assume there's no difference — that the true conversion rates of the two screens are identical. The reason for assuming this is that it makes the math possible: there are infinitely many ways a difference could be some size, but exactly one way for it to be zero.
2. Compute the probability of the observed difference under that assumption. That probability is the p-value, the chance of a difference this large arising from randomness alone.
3. Compare against a threshold fixed in advance. 0.05 is the convention. A p-value of 0.03 falls below it, so chance becomes hard to credit as the explanation.
4. Record the judgment. Significant does not mean the difference is large — it means chance is a poor explanation for it.

There's no special basis for 0.05. It's a hardened convention. It implies that in twenty experiments, one will cross the line with no real effect present, which is why teams running many experiments either lower the threshold or reconfirm the same result.

Sample size moves this calculation heavily. The same difference comes out significant with a large sample and not with a small one. At 200 per group, 4.8 percent against 5.2 percent is indistinguishable from noise; at 50,000 per group, the same gap is significant. So before starting, you compute how many observations a conclusion requires — that figure is the sample size. Statistical power, the probability of catching a real difference when one exists, is the other half of that calculation.

Fix the stopping point in advance too. Watching results and stopping the moment they turn significant means stopping at whichever moment chance crossed the line. Writing down the duration and the sample first, then waiting, is a condition of using this standard at all. Peeking daily and stopping on the first significant reading roughly triples the rate of false positives.

Four settings put this judgment to work.

1. An A/B test, splitting traffic randomly between two versions and comparing. The most common setting, and the basis for deciding whether a new screen ships to everyone.
2. Deciding whether a metric moved. When this week's churn is 0.4 percentage points above last week's, this is what decides whether to investigate. Inside the band the number wobbles in weekly, there's nothing to find.
3. Reading survey results. With eighty respondents, this decides whether a difference between two groups gets reported at all. With a small sample, a visible difference still sits inside the range of chance.
4. Designing the experiment. The standard applies before results exist, not only after. Decide what size of difference you want to catch and the required sample falls out — and if you can't collect that sample, the better decision is not to run it.

One more thing has to be read alongside: the size of the difference. At 100,000 per group, 4.80 percent against 4.83 percent can come out significant, but whether 0.03 percentage points is worth changing the screen for is a separate judgment. Significance answers whether it's chance; it says nothing about whether it's worth doing.

"There's a difference but it isn't significant. Running it two more weeks."

Deferring the judgment because the sample is short.

"It's significant and the difference is 0.03 percentage points. Is this worth shipping?"

Separating significance from magnitude.

"Don't stop when you peek. We said two weeks, so it runs two weeks."

Explaining why the stopping point was fixed in advance.

At Pagedeck, Mia writes down three things before starting an experiment: the difference she expects, the sample required to catch it, and when it stops. Last quarter a signup-screen experiment came out needing 40,000 per group, which divided by daily traffic worked out to three weeks.

That calculation also killed an experiment. The enterprise pricing page gets 300 visitors a day, so catching the expected difference would have taken four months — and nothing else could be held still for four months. So that screen got handled by talking to five users instead, which answers a different question less precisely and does it in a week.

The most common error is reading a non-significant result as no difference. The threshold wasn't crossed, so the effect looks absent. But what the result tells you is only that the data you collected can't separate the effect from chance. With a small sample, a real effect produces exactly this outcome. Concluding there's no effect requires separate evidence that you collected enough to have caught an effect of that size, which is exactly what power is for.

The other misreading is treating significance as importance. The word sounds like it means important. It only means chance doesn't explain it, and it says nothing about magnitude — with a large enough sample, a tiny difference comes out significant. So results get reported with both the significance and the size of the difference, and the one that usually decides the action is the second.