Control Group

For a long time a number going up was treated as proof that a change worked. Conversion moved from 9 percent to 12 percent in the week the sign-up copy was rewritten, so the copy got the credit. The trouble is that the copy was never the only thing that changed that week. Ad spend went up, the school term started, and a competitor had an outage.

Pagedeck lived through exactly this. Ravi reworked the sign-up screen, the numbers jumped the following week, and he reported it as a win. Then Grace asked whether that wasn't also the week the team doubled ad spend. Nobody in the room could answer. There was no way left to say how much of those three points belonged to the copy.

What was missing wasn't data. It was something to compare against. The team had the number after the change and nothing showing what would have happened without it. A control group manufactures that missing number. Set aside a group of people who live through the same weeks under the same conditions but never receive the change, and their result is what "no change" looks like.

Split incoming users into two groups, show the change to only one of them, and measure both over the same window. The group that doesn't get the change is the control, and the gap between the two numbers is what the change is worth.

Three conditions have to hold.

1. Same period. Comparing this week's new screen against last month's old screen drags every difference in season, advertising, and day-of-week mix into the result. Both groups have to run on the same days at the same hours.
2. No skew in the split. Dividing by sign-up order or by the last digit of an account ID piles older users, or a batch of accounts one company created at once, onto one side. The procedure that handles this is randomization, and something close to a coin flip is what makes the two groups resemble each other on average.
3. Same ruler. Both groups get the same conversion definition and the same counting window. If one side counts completed payments and the other counts payments started, the difference tells you nothing about the change.

With those three in place, the control group stands in for the counterfactual — the outcome in the world that didn't happen, the number these people would have produced without the change. Nobody can live through both worlds at once, so a group with matching characteristics is put in that spot instead.

Size is decided in advance too. A control group that is too small can't separate a real difference from noise. Calculate the sample size before launch and convert that figure into days, so the experiment has a length and not just a target.

The most common home is the A/B test, where the untouched variant is the control by definition. A multivariate test that compares combinations of several elements still keeps one combination with nothing changed, and that one holds the baseline.

The idea travels well beyond screens. Withhold the onboarding email from two out of every ten new accounts and you can tell whether the email actually lifts activation. Turning ads off in selected regions and comparing them against the rest has the same shape, except the control is a set of regions rather than a set of people. Sales teams do it too when a new price sheet goes to part of the customer base while the rest keep the old one. The unit being split changes from one setting to the next; the role it plays does not.

The hard call is the split ratio. Fifty-fifty is statistically the most efficient, but holding half the traffic on the old screen feels wasteful when the change looks obviously better. That's why teams sometimes tilt the split to 90/10 — which shrinks the control and makes the same difference take far longer to confirm. Put the minimum detectable effect (MDE) next to the ratio, work out the resulting timeline, and decide from there.

"Without a control group this is a before-and-after trend, not an experiment result."

The number did move, but the claim that the change caused it isn't being accepted.

"Let's keep the control at 10 percent this time. That pushes the call out to three weeks."

Spelling out the trade: a smaller baseline group buys traffic for the new version and costs time.

"The traffic sources look different between control and treatment."

The two groups that were supposed to be randomly split don't actually match, so the assignment needs checking.

A checkout experiment at Pagedeck runs like this. Theo configures the new screen to appear for half the traffic, and before anything starts Mia writes down what counts as a conversion and which date the team will judge on. Two weeks later the control converts at 11.2 percent and the treatment at 12.8 percent.

That 1.6-point gap is where the discussion begins. Both groups saw the same ads in the same weeks, so ad spend never comes up. The only open question is whether a gap that size could appear by chance, and that one has a calculated answer. Had the control landed at 12.9 percent instead, the same two weeks would have ended the discussion just as cleanly, and the team would have kept the old screen without anyone needing to argue about it.

People often treat the control group as a cost imposed on its members, who are stuck with the worse screen. The assumption underneath is that the change is already known to be better. Run enough experiments and you find that a large share of them come back flat or negative. Not knowing which version wins is the reason the experiment exists, and the control group is what stops a bad change at half the traffic instead of all of it. Seen that way it costs less than shipping to everyone and finding out a month later.

Substituting last month's numbers for a control is the other common shortcut. The data already exists and nobody has to be held back, which makes it attractive. But between last month and this one, the seasonality, the day mix, the campaigns running, and the competitive picture all moved along with the change. Read that combined shift as the effect of one edit and you ship a change that does nothing, then plan the next quarter on top of the inflated figure. Month-over-month numbers still have a job — they show what shifted. They just can't say what caused it.