Sample Size

Experiments used to run for a few days, and whichever side was ahead won. Pagedeck did exactly that with the wording on its signup screen: results were read on day three, Ravi reported conversion up from 3.1 to 3.6 percent, and the new wording went out to everyone.

A week later the same screen converted at 3.0 percent. About 900 people had landed in each group over those three days, and at that size two identical screens will happily produce 3.1 and 3.6. Twenty-seven signups is 3.0 percent and 32 signups is 3.6 percent. The gap was five people.

After that happens a few times, nobody believes experiment results at all. A test wins, and a month later the change gets rolled back. Sample size is the value that moves this problem to the front of the process. Before starting, you ask whether the traffic you can gather is enough to tell a difference of the size you care about, and if it isn't, you either don't run the experiment or you change its terms.

Four inputs go into a formula that returns the head count per group: the current conversion rate, the smallest difference you want to detect, the significance level, and statistical power. A calculator or analytics tool does the arithmetic, so the actual work is deciding the four values.

1. Check the current rate. For a signup screen that might be a four-week average of 3.0 percent. The lower this rate, the more people you need.
2. Decide the smallest difference worth detecting — moving 3.0 percent to 3.6 percent, a relative improvement of 20 percent. That value is the minimum detectable effect (MDE).
3. Set the significance level and the statistical power. Convention is 5 percent and 80 percent. Power of 80 percent means that when a real difference exists, you have an 80 percent chance of catching it, so one time in five a real difference slips past.
4. Divide the head count by daily traffic to convert it into a duration. A number of people, on its own, never makes it into a schedule.

Those four values come out to roughly 13,000 people per group. That's 26,000 across both, and at 1,000 visitors a day to the signup screen it's a 26-day experiment.

The relationship between step 2 and the head count is worth sitting with. Halve the difference you want to detect and the people you need doesn't double — it quadruples. Asking for a 10 percent relative improvement instead of 20 percent means 52,000 per group, and the 26-day experiment becomes a three-month one. So step 2 gets settled with a product question rather than a statistical one: if the difference turns out to be this small, are we really going to change the screen?

A good sample size isn't a small one; it's one that converts into a duration the team can live with. The working rule of thumb is four weeks. Run longer and the season changes underneath you, other releases ship into the same window, and afterwards nobody can say what produced the difference.

When the duration comes out too long, change the terms rather than hunting for more traffic. Three moves are available. Widen the difference you're trying to detect. Widen the audience — an experiment limited to new signups doubles its daily traffic once returning visitors are included. Or move to a metric further up the funnel: if final payments run 30 a day while entries to the payment screen run 400, design the experiment on the second number and treat payments as context.

An experiment started without this calculation can't be interpreted at all. When it shows no difference, there's no way to separate two very different meanings — that the two versions really perform the same, or that the groups were too small to reveal a difference that was there. A result from an A/B test with 300 people per group carries no information either way. Writing the number down in advance splits those apart. Reach it and find nothing, and the conclusion is that no difference of the stated size exists. Fall short, and the experiment simply has no conclusion.

"Did you work out the sample size? At 900 visitors a day this needs three weeks."

A check on whether the duration was calculated before starting, because that duration has to sit in the schedule.

"Detecting a one-point difference takes 25 times the sample. Let's set it at three points."

The design aims at a difference so small that the required run time is unworkable, so the bar should be raised.

"We can't stop early just because we're ahead — the sample isn't full yet."

Calling a winner before the planned head count arrives means reading a random gap as an effect.

At Pagedeck this value sits in the first field of the experiment proposal. When Ravi proposes a test, the first thing Mia checks isn't what's being changed but the sample size and how long it takes to fill. Anything that finishes inside four weeks gets approved as written; when the answer comes back at eight weeks, they look together at whether the audience can be widened or the metric moved further up the funnel. If neither works, the question goes to user interviews instead of an experiment.

The most common assumption is that a bigger user base requires a bigger sample. Half-remembered talk about populations in opinion polling makes the connection easy to draw. But what sets the head count is the current conversion rate and the difference you want to see, not the total size of the service. A product with 50,000 monthly visitors and one with 5 million need almost the same number of people to detect the same difference at the same 3 percent conversion rate. All that differs is how long collecting them takes. Left uncorrected, this belief leads small teams to accept thin samples and pile up conclusions with nothing under them.

The second frequent mistake is topping up the sample when the result comes back ambiguous. A few more days feels like it should sharpen the picture, so the instinct is understandable. But extending the run while watching the result voids the calculation. Both groups keep fluctuating, so the longer you stare, the more certain it becomes that one side will be ahead at some moment. Wins collected that way are the ones that get rolled back next month. This is why the stopping point gets written down alongside the sample size, before anything starts.