Peeking Problem

This problem arrived with live dashboards. Results used to be collected after a run finished and calculated once, so there was no way to look partway through. Now opening a dashboard shows who is winning as of today.

Which produces a conversation on day three. B is ahead, so hasn't it already won, and why wait another eleven days? The experiment gets stopped, B ships to everyone, and a month later the metric is back where it started with nobody able to explain why.

The cause is that numbers wobble daily. Even with no real difference between two versions, one side pulls ahead by chance on some days and the order reverses on others. Flip a coin 100 times and heads settles near 50, but there is very often a moment around flip 20 where heads has come up 13 times. Look every day and you will meet one of those stretches, and stopping there writes down a coincidence as a finding. The finding is sticky, too: once it ships, next quarter's plan assumes it was real.

The peeking problem is the name for that trap. What causes it isn't looking. It's stopping because of what the look showed.

Applying a decision rule many times lets the chance of passing it by luck accumulate with every application. A calculation designed to be run once is being run twenty times, which loosens the threshold itself.

Experiments get judged by calculating the p-value, the probability of seeing a difference this large from chance alone, and deciding that chance is an unlikely explanation once that value drops below a line set in advance. Setting the line at 5 percent is conventional, and the whole judgment is called statistical significance. Five percent means the chance of claiming a difference that isn't there is one in twenty.

The catch is that this is the probability for looking once. Look once a day for twenty days and stop the first time the value dips below the line, and the chance of a false claim is no longer 5 percent — it rises past 60. Run the experiment with two identical versions and six times out of ten a winner appears.

Five things prevent it.

1. Fix the number of observations before starting. Calculating the sample size, the minimum number of observations needed to conclude anything, belongs here. Ten thousand per arm is a different experiment from one thousand, and that number has to be settled before anyone has seen a result.
2. Fix the day of judgment. Until that date arrives, no conclusion gets written no matter how the numbers move.
3. If judging partway is genuinely needed, switch to a method built for it. Approaches exist that tighten the threshold in advance for repeated looks, so twenty checks still leave the false-claim rate at 5 percent. The cost is a longer run, because a tighter threshold means waiting for a larger gap.
4. Hide the interim winner on the dashboard. What's visible starts conversations, so screen design holds better here than a rule does.
5. Write the stopping condition down before launch. With a sentence already stating what ends the run, there's nothing to negotiate on the day.

The mirror image is the same error. Delaying a conclusion because you're losing, and running on in hope, is also choosing the moment of judgment based on the result.

There are four places this comes up.

The first is the operating rules for an A/B test, where two versions are shown to randomly split groups and compared. Most of the reason teams write down when a test starts and when it gets judged traces to this problem.

The second is the meeting where results get reviewed. One question gets added: how many times was this looked at, and was the stopping day the day that had been planned? With no answer to that, the answer matters more than the numbers do.

The third is building the dashboard. Whether an experiment screen displays today's winner is decided differently by a team that knows about this and a team that doesn't.

The fourth is defining the exceptions. Error rates and payment failure rates get watched daily and killed immediately when they worsen. That isn't a judgment about which version won; it's a guard against an incident, so the same rule doesn't apply to it. Writing the two lists separately — metrics that decide and metrics that halt — is what keeps the exception from swallowing the rule.

Setting it beside the other risks in the same branch makes the boundary clearer. The novelty effect, where something looks good briefly because it's new, is an error created by too short a window. Sample ratio mismatch, where assignment came out off the intended split, means the experiment itself is broken. The peeking problem is the case where the experiment is fine and the moment of stopping was chosen to fit the result.

"It's day three and we're already winning. Can't we stop now?"

The moment to check whether a stopping date exists.

"The winner flipped every time I opened this."

A signal that the difference sits inside the range of chance.

"Let's calculate the sample size before we start."

Settle when to stop before anything launches.

At Pagedeck, Ravi ran an experiment on the signup screen, saw good numbers on day four, and proposed ending it early. What Mia asked on the spot was how many times he had opened the results, and the answer was daily.

So the experiment ran its planned two weeks and the final result came back as no difference. The two of them then changed the experiment dashboard to hide the winner until the end date and show only the observations remaining.

The most common mistake is believing a large enough sample makes interim looks safe. More data means less wobble, so the reasoning follows. The wobble does narrow, but this problem comes from the number of judgments rather than from size. On a product with 100,000 daily visitors, twenty looks inflate the false-claim rate exactly the same way. Miss the distinction and the largest teams stop early most often, and reversed decisions pile up quarter by quarter.

The second is reading this as a ban on looking. The explanation above sounds like one. But looking and stopping are different acts. Whether the experiment is running correctly, whether assignment came out at the intended split, whether errors have climbed — all of that has to be checked daily. What's off limits is settling the winner with those numbers, and if settling it partway matters, the fix is a calculation built for repeated looks rather than a promise not to look. Teams that get this right look constantly and decide once.