Treatment Group

For a long time, the way to check whether a screen change worked was a before-and-after comparison. Change the signup button copy, then compare next month's conversion rate against last month's. Last month 8 percent, this month 9.5 percent, call it an improvement.

Except the button copy wasn't the only thing that changed over that month. The ad budget went up, a competitor raised prices, the season turned. When someone in the review asks whether it was the extra ad spend, there's no way to answer. The reverse case is just as awkward: if the copy genuinely helped but a major outage that month pushed conversion down, a good change gets reverted.

A treatment group removes the problem by creating something to compare against in the same period. Same weeks, same advertising, same season — one group sees the new copy and another keeps the old one. The only remaining difference between the groups is the copy, so a difference in results can be attributed to it. All that changed is that the comparison moved from two points in time to two groups at one point in time.

Five things have to be decided when setting up a treatment group.

1. What changes — one thing at a time. Change the button's color, copy and position together and a difference in results tells you nothing about which of the three caused it. Wanting all three means running three experiments.
2. Who goes in — randomization decides, the procedure for splitting groups without bias. Split on signup order or the last digit of an account id and other properties ride along: early signups cluster the earliest users, and one range of ids may be accounts one company created in a batch.
3. How many go in — half and half is the default but not a requirement. A change that's hard to reverse starts at 10 percent. Shrinking one side lengthens the experiment, though. A 90-to-10 split takes more than twice as long as an even one to reach the same conclusion.
4. How long it runs — calculate the sample size first, the count you need before a conclusion is possible. With 8,000 needed per group and 1,000 visitors a day, an even split still makes it a 16-day experiment. That number gets fixed before the start, and nothing gets judged before it.
5. Whether it actually rendered — count how many of the people assigned to the treatment group saw the new screen. Assign 5,000 and log 4,100 exposures and 900 people saw the old screen somewhere, which makes any conclusion understate the effect.

The split ratio going wrong is itself a signal. An even split that arrives as 52 to 48 means something went wrong in assignment, and the results get discarded.

The most common setting is an A/B test — showing two versions to randomly split audiences and comparing outcomes — where the side seeing the new version is the treatment group. Screen copy, how prices are displayed, the number of signup steps all get decided this way.

Staged rollouts use the same structure. Before turning a feature on for everyone, turn it on for 10 percent and watch error rates and metrics. Clean, then 50 percent, then everyone. That 10 percent plays the treatment group's part, with a slightly different purpose: the aim isn't measuring an effect but keeping an incident small. So in this case nobody calculates a sample size — only a duration and an error-rate threshold.

The reason a short run is a bad idea also lives on the treatment side. The novelty effect is the temporary lift that comes from something merely being new. Change a screen and existing users click the thing that moved, so the treatment group's numbers jump for the first few days. They commonly settle back within two weeks, so declaring victory on three days of data pushes a change that gains nothing to everyone.

Some changes go out with no treatment group at all: screens a regulation requires changing, and cases where showing different people different values causes confusion. When one person on a team sees new pricing and their colleague sees the old, the support question arrives before any result does.

"Treatment gets the new copy, control stays as-is, and we run it two weeks."

Stating an experiment's design in one sentence.

"Treatment assignment is 5,000 but exposure is 4,100. We need to find the leak."

Catching a gap between assignment and actual rendering.

"This one's hard to reverse, so I'm starting the treatment group at 10 percent."

Sizing the treatment group to the risk.

At Pagedeck, Mia owns experiment design. Last quarter she put the new signup screen Noa built in front of a 50 percent treatment group. One thing changed: signup went from three screens to two. Copy and color stayed exactly as they were.

Two weeks later the treatment group converted at 11.2 percent against control's 9.8. Before calling it, Mia checked three things: whether the head count in each group matched the intended ratio, whether new-screen exposure inside the treatment group exceeded 99 percent, and whether the difference held over the period excluding the first three days. All three passed, and the change went to everyone.

The most common mistake is thinking a larger treatment group makes the result more accurate. More people does stabilize numbers. But the accuracy is set by whichever group is smaller. Put 90 percent in treatment and leave 10 percent in control and the control numbers swing too much to read a difference. Growing the treatment group requires growing control with it, and when visitor volume is fixed, an even split is the fastest. Making one side small is a trade that gives up accuracy to reduce risk.

The other mistake is reading a rise in the treatment group's metric as success. The changed side went up, so it sounds like it worked. But what matters is the gap against control, and whether that gap is too large to be explained by chance. Treatment going from 9.8 to 11.2 means nothing if control went from 9.8 to 11.0 in the same weeks. And watching only the target metric hides damage elsewhere. If cutting signup steps raised conversion while the share of users still present a month later fell, the experiment was not a success.