Statistical Power

A result reading "no difference" mixes two cases together: there genuinely was no difference, and there was one and the experiment failed to catch it.

Ravi once changed the copy on the signup screen and ran an A/B test. Two weeks in, the result wasn't significant, so the copy was declared ineffective and the team moved on. An A/B test shows two versions to randomly split groups and compares the results.

Six months later the same copy went back up and the result came out differently. Traffic had grown fivefold in the interim, and a difference of the same size was now significant. The effect had been there all along; the first experiment was not an instrument capable of seeing it. The conclusion drawn from it had been wrong for six months.

What was wrong with the first experiment was the design, not the conclusion. With that sample, the probability of detecting a difference of that size was something like 30 percent — and nobody had calculated that before running it. Statistical power is the figure you measure in advance to avoid exactly this. It converts a vague worry about sample size into a number you can act on.

Power follows once four quantities are set. Fix any three and the fourth is determined.

1. Significance level. The ceiling on the probability of claiming a difference that isn't there, usually set at 5 percent.
2. The size of difference you want to detect. Whether you need to see a conversion rate move from 4 percent to 4.8, or only if it reaches 6. This quantity is the minimum detectable effect.
3. How much the values vary. How widely results spread under identical conditions. For rates like conversion it's computed from the baseline rate.
4. Sample size. How many go into each group.

In practice nobody calculates power; they fix power and solve for sample size. Hold power at 80 percent and significance at 5 percent, and for a test comparing rates the count needed per group works out to roughly 16 times the baseline rate's variability divided by the square of the difference.

Putting numbers in shows the scale. With a baseline of 4 percent and a target of detecting 0.8 percentage points, each group needs 9,600 people — 19,200 across both. At 800 visits a day that's 24 days, and half that traffic doubles it.

One property stands out here. Halve the difference you want to detect and the sample needed quadruples. Looking for 0.4 percentage points takes 38,400 per group and 96 days. Because confirming small differences gets expensive fast, experiment design usually starts by deciding the smallest difference worth seeing. That decision is a product judgment rather than a statistical one.

The convention is 80 percent: if the difference is real, four out of five runs catch it and one misses. It isn't an absolute standard, and it moves with the cost of missing.

1. Small reversible changes. Eighty percent is fine. Miss it and you try again later.
2. Experiments deciding a large investment. Raise it to 90 or 95 percent. Going from 80 to 90 costs roughly 30 percent more sample.
3. Judgments involving safety or regulation. The cost of a miss is high, so set it higher.

Below 50 percent there is barely a point in running the experiment, because the outcome turns on something close to a coin flip. A "no difference" result in that state isn't evidence of no difference; it means the experiment failed to answer, and the honest report says so rather than reporting a null.

Low power has a less familiar side effect. In a small experiment, reaching significance requires the observed difference to be very large. So significant results from small experiments are likely to overstate the real effect. A 40 percent conversion lift measured on a sample of 200 usually shrinks when the test is run properly, which is why early wins so often fail to reproduce.

Using this figure in practice means computing the sample size before starting and converting it into a duration. Sample size is the minimum number of observations needed to reach a conclusion. If the arithmetic returns four months, that experiment cannot be run, and the response is either attempting a larger change or confirming it some other way.

"This test has 30 percent power. Even a null result tells us nothing."

The design's detection probability is too low for the result to be interpretable.

"Seeing 0.4 percentage points needs 80,000 across both groups. That's three months."

The cost of resolving a small difference is beyond what's affordable.

"They report a 40 percent improvement on 200 people. Hard to take at face value."

A significant result from low power is likely inflated.

Pagedeck runs this calculation before starting an experiment. The rule came out of Ravi's copy experiment: compute the duration needed at 80 percent power, and if it exceeds two weeks, don't run that experiment.

Two paths handle the rest. Changes expected to have a large effect get made larger, widening the difference; small improvements ship without an experiment. Mia attached one condition — anything shipped without an experiment has to be reversible. Skipping the test is acceptable only where being wrong is cheap.

The most common mistake is reading "not significant" as "no difference." The phrase sounds like a verdict of no effect. But two cases produce that result, and without knowing the power you cannot tell them apart. In an experiment with 30 percent power, a null result is common even when a difference exists. Reporting power alongside the result reduces this confusion considerably, because a reader can then see what the experiment was capable of.

Assuming more sample always fixes it is also frequent. The formula is written in terms of sample size, so it looks that way. But sample takes time to accumulate, and a long window lets the product or the season change mid-experiment until the comparison blurs. So the alternatives get considered too: widen the difference you're looking for, switch to a metric that varies less, or narrow the population to a group likely to respond strongly. Each of those buys detection without buying time.