p-value

In an A/B test, where two options are shown to randomly split groups and the results compared, the standing problem is whether a higher number means anything. Flip a coin a hundred times, get 52 heads, and the coin isn't weighted. Numbers wobble up and down even when nothing is different.

Pagedeck got caught by that wobble. Ravi reported that changing the signup button copy had lifted conversion from 12.4 percent to 13.1 percent. Three days later, rerunning the same test against a different set of weekdays put the two options in the opposite order. Each option had held 300 people. It was the difference between 37 and 39 conversions out of 300, a state where which side two people happen to land on flips the conclusion.

A p-value measures the size of that wobble. It assumes a world with no difference whatsoever and calculates the probability that even that world would produce the difference just observed. Under statistics, the basic concepts used when working with data, it belongs to the testing group, the one that weighs whether an observed difference is chance.

It isn't a number anybody computes by hand; analysis tools report it. What's worth knowing is what goes into that computation.

1. Assume there is no difference. Setting the two conversion rates equal is the assumption, called the null hypothesis. A p-value is always calculated under it.
2. Measure the observed difference. Between 12.4 percent and 13.1 percent, that's 0.7 percentage points.
3. Find the probability of a difference that large arising while the assumption holds. Three ingredients feed that calculation: the size of the difference, the sample size, and how spread out the values are.

Of the three, sample size swings the result most in practice. The same 0.7 percentage-point difference produces a p-value near 0.6 at 300 people per side and drops below 0.01 at 30,000. The difference didn't change and the verdict flipped, which means at 300 that difference can't be told apart from chance and at 30,000 it can.

There's one way to read it. A p-value of 0.03 means that even in a world with no difference, a result this large turns up three times in a hundred. That's rare enough to conclude a difference is the better bet.

Being clear about what it isn't helps too. It is not the probability that option B is better, 97 percent. It isn't the probability that an assumption is true; it's the probability of data like this arising if the assumption is true. The direction is reversed, so the two sentences can't be swapped.

The conventional line is 0.05. Nothing special justifies that value. It's an agreement to accept catching something imaginary roughly one time in twenty, and fields set it differently. Clinical research where lives are at stake uses a far lower line, and some teams choosing button copy set it at 0.1 and decide faster.

What has to be honored is the order. Set the line before the test and don't move it after seeing the result. Shifting the line to 0.1 because 0.07 came back turns a judgment into an after-the-fact explanation.

Sample size gets set in advance too. Decide the statistical power — the probability of catching a difference when one genuinely exists — and calculate the headcount that requires. A test begun without that calculation produces results nobody can read: there's no telling whether "no difference" means there really is none or that the sample was too small to see it.

Not looking at the p-value alone matters as much. Gather 30,000 per side and a 0.1 percentage-point difference comes back at 0.01. Statistically it isn't chance; commercially there's no reason to act. So the size of the difference and the confidence interval, the range the true value is thought to sit in, get recorded alongside it.

Not stopping partway is the other rule. Check the p-value daily and end the test on the day it dips below 0.05, and the chance of declaring a difference that doesn't exist multiplies, because that value wanders up and down on its way.

"Three hundred per side and a p-value of 0.6. This number still says nothing."

The sample is too small to separate the result from chance.

"The p-value is 0.01 and the difference is 0.1 percentage points. Is that worth shipping?"

Separating the statistical judgment from the commercial one.

"We set the line at 0.05 before starting, so 0.07 isn't a pass."

Refusing to move the standard after seeing the result.

In Pagedeck's experiment review, Ravi brings the numbers and Mia asks three things in order: whether the sample reached the planned size, whether the planned duration ran out, and whether the size of the difference is worth acting on. If the first two aren't satisfied, the p-value doesn't get read.

What changed after the two of them started reading numbers together was the count of experiments. Six a month became three. What disappeared was conclusions reversing three days later.

The most common reading is that a lower p-value means a larger effect, since a smaller number carries the impression of being better. But this value doesn't measure the size of an effect. It measures how hard it is to explain the observed difference as chance. Push the sample high enough and even a tiny difference carries a low p-value. So the two always get read together: how much difference there is comes from the size, and whether that difference can be trusted comes from the p-value.

The other frequent move is concluding there's no difference when the line isn't crossed — writing that two options are the same because 0.07 came back. What the value actually reports is only that this data can't separate the result from chance. That's an absence of evidence rather than evidence of absence, and the two are different. Double the sample, rerun, and the same difference might come back at 0.02. Teams that keep this distinction write "not confirmed in this sample" in the report instead of "no difference."