Holdout

Teams that run experiments diligently kept hitting the same strange result. Twenty tests in a year, twelve of them winners, each written up with a conversion lift somewhere between 2 and 5 percent. Then someone would put January's conversion rate next to December's and find them nearly identical.

Quarterly reviews got awkward in a predictable way. The presenting side cites individual test results as evidence of progress; the listening side looks at the company dashboard and asks why that number hasn't moved. Both are reading real numbers and the two accounts never meet. With no way to answer, the conversation usually drifts toward blaming measurement or the market.

The cause sits in how individual experiments end. Once a test concludes, the control group gets released and sees the new screen, and from that moment nobody in the product is still on the old one. The next test starts from the already-changed screen, so the baseline is rebuilt from scratch every time. A holdout is the idea of leaving one baseline standing for a long while. If some people are still looking at last January's product, the difference between then and now can be measured directly.

Pick a random slice of all users and leave them on the old experience even after each experiment ends. Everyone else receives every winning change along the way.

Running one takes five steps.

1. Set the size, usually 1 to 5 percent of all users. These people don't get any of the improvements, so a larger slice costs more; a smaller one loses the statistical power to detect the difference you're looking for.
2. Draw the group once through randomization and then freeze it. Redrawing per experiment resets the baseline every time and nothing accumulates. Splitting by signup order or by region doesn't work either, because those criteria drag other properties along with them.
3. Choose a duration. Some teams run quarterly cycles, others hold for a full year. The more changes that will stack up inside the window, the more there is to read at the end.
4. Ship every A/B test winner to everyone except this group. Ten features go out and this group's screens never change once.
5. At the end of the window, compare the core metric across the two groups. That gap is the net lift of the entire year, added up.

When the window closes, release the group and draw a fresh one for the next cycle. Holding the same people indefinitely means their experience falls further behind every quarter, their churn rises, and the comparison itself gets distorted.

One calculation has to happen up front. If the holdout is 1 percent of users, check that the group clears the sample size you need. A million monthly visitors gives you 10,000 people at 1 percent, and around a 3 percent conversion rate, 10,000 people can reliably detect a difference of roughly half a percentage point or larger. If the year's accumulated gains are smaller than that, the holdout will end without seeing anything at all.

Recommendations and personalization are the most common home. A ranking model changes dozens of times a year, each change shipping with a small documented gain, and no individual test can tell you whether those small gains actually added together. So a slice of users keeps seeing an unpersonalized default list, and the two are compared twelve months later.

Marketing channels use the same structure. Turn ads off for one region or one user segment, compare against everyone else, and you've stripped out the revenue that would have appeared without the advertising. This is the standard way to measure incrementality. Since it means accepting lower revenue while the ads are off, these runs are usually kept to a month or two.

The third setting is annual reporting. When a product team has to answer, in one number, how much it improved things over a year, this is where that number comes from. Summing up individual test results always overstates it, and teams that put the holdout number alongside keep their credibility longer.

"This year's holdout came back. The tests summed to 18 percent; the real gap was 6."

The sum of the wins looks very different from the actual improvement.

"A 1 percent holdout won't see it. We need 3 percent to detect a difference this size."

The group is too small for the comparison to reach a verdict.

"Does this feature go into the holdout too? It's a legal requirement."

Confirming that changes everyone must receive can't be withheld from this group.

At Pagedeck, Mia froze 1 percent of all teams into a holdout last January. Over that year fourteen things changed — document recommendations, search result ordering, the layout of the home screen — and nine of them were winning tests. Added together, those nine predicted a 21 percent lift in weekly active users.

The measured gap in December was 7 percent. Mia reported that number as a baseline rather than a bad grade. Tracing back which changes had actually contributed, three of the nine turned out to be counting the same behavior by the same users under different names, and two were novelty effects that had faded within the first two weeks. The next year's plan changed how both kinds of test get called.

The common expectation is that the holdout result should land somewhere near the sum of the individual wins. If each test was designed properly, the effects ought to add up — a reasonable thing to assume. In practice the holdout usually comes in under half of the sum, for three reasons: separate experiments double-counting the same improvement, effects that sparkle for a few weeks and disappear, and the bias that comes from only the favorably random results getting declared winners. The mismatch isn't a malfunction. It's the thing the holdout was built to reveal.

Reading the group as pure cost and proposing to remove it is the other frequent move. It's true that 1 percent of users going a year without improvements shaves something off revenue on paper. What that buys is the only honest answer to whether the last year of work was worth anything. Teams that drop the holdout go back to reporting the sum of their test results, and the people receiving that report tend to notice the inflation before the team does.