Randomization

Comparing two versions requires the two groups seeing them to resemble each other. If they do not, there is no way to tell whether a difference in the result came from the version or from the group.

Because splitting is fiddly, convenient criteria got used a lot. Half by earliest signup against half by latest, or odd account IDs against even ones. The problem is that such criteria drag other properties along with them. The earliest signups are crowded with people who sought the product out, and they convert well no matter what you show them. A particular range of IDs may be accounts one company created in a single afternoon.

Splitting by time produces the same failure. Show A from Monday to Wednesday and B from Thursday to Saturday and the day-of-week effect is baked into the result.

Randomization removes the problem by splitting on something close to a coin flip. Its value is not that it balances the properties you can see. It is that properties you do not know exist get spread evenly across both sides. Matching acquisition channel and device and usage habit one at a time never ends, but a random split scatters the things that were never on your list. It protects against a confounding variable — a hidden factor distorting the relationship — without your having to know about it first.

Fix the unit of assignment, split on a value nobody can anticipate, and check afterward that the groups actually resemble each other.

1. Fix the unit of assignment. Per person or per visit comes first. Split per visit and the same person sees A on Monday and B on Tuesday, which feels like the product changed under them. For a product teams use together, move up one level and split by team.
2. Fix the ratio. Half and half is the default. For a change that is hard to reverse, start at 90 to 10, confirm nothing is broken, and widen from there. The 10 percent side still has to be large enough to detect anything, which is a separate calculation.
3. Fix the method. Hashing a user identifier into buckets by a fixed rule is common. The same person has to land in the same group whenever they return. Draw fresh each time and one person will cross between groups.
4. Fix the moment of assignment. Assign when someone actually reaches the surface in question. Pre-assign the whole user base and people who never reach that surface get counted into the groups, which buries the difference.
5. Check after splitting. A 50-50 split arriving as 54-46 means something is wrong in the assignment or the counting. Compare the plan mix and the signup-date distribution across groups as well. When those diverge badly, stop and find the cause first.
6. Do not touch it mid-flight. Changing the ratio or the assignment rule partway blends the earlier period with the later one. If the ratio has to change, the honest move is to stop and start a new run.

It fits where user volume is sufficient and splitting at random causes no harm: layout, copy, recommendation order, notification timing. The two resulting groups become the control group and the treatment group, and the A/B test that follows runs on top of this assignment.

Four limits are worth knowing.

1. Some things cannot be split. An experiment selling the same product at two prices becomes a trust problem once users notice, and a brand campaign that goes out once to everybody has nothing to divide. With twenty enterprise customers, a split leaves ten per group and no conclusion is reachable.
2. The groups can affect each other. If the inviter is in treatment and the invitee is in control, the effect of a new invite screen shows up in the control group's numbers too. Those cases get split by a unit that does not meet itself, like team or region.
3. A small sample skews even when random. Split twenty accounts by coin flip and the large customers land on one side. Which is why the sample size gets calculated first, and an experiment that cannot secure the time to collect it does not start.
4. It does not remove every bias. A bug that only appears in treatment, or a screen that is slower for one group, ruins the result regardless of how well the split was done. Whether the assignment worked and whether the experiment is running cleanly are checked separately. A clean split is a necessary condition, not a sufficient one.

"We can't split on the last digit of the account ID. That range might be one company's accounts."

Pointing out that a convenient criterion drags other properties along.

"Let's split by team. People on the same document seeing different screens can't explain anything to each other."

A judgment that the unit of assignment has to move up a level.

"It's a 50-50 split and it came in at 54-46. Start with the counting."

Confirming the assignment did what it was supposed to before anything else.

At Pagedeck, Mia defaults to team as the unit whenever she opens an experiment. It is a product for writing documents together, so a split inside one team generates support tickets. Theo pinned the assignment rule so that a given team stays in the same group for the whole run.

One experiment failed on this. A change to the invite screen was split per person, and because treatment sent the invitations that control received, both groups' numbers rose together. The result read as almost no difference; rerun with a split by team, the difference was clear. The same change produced opposite readings depending only on the unit.

The most common error is treating a regular criterion as equivalent to random. The last digit of an ID or the order of signup was not chosen by us, so it looks fair. But the requirement for randomness is not that we refrained from intervening — it is that the criterion is uncorrelated with everything else. Signup order is strongly tied to usage habits, and ID ranges are tied to how accounts were created. So a difference obtained this way cannot be read as an experimental result, and a change shipped on that basis leaves nothing to reverse it with.

The second error is believing that randomizing is what makes something an experiment. Splitting is the first and most visible step, which invites the conclusion. But it is one component. How many people to collect, the minimum detectable effect describing the smallest difference this run can catch, what counts as success, and when to stop all have to be settled too, or a beautifully split pair of groups yields nothing. For effects over a longer horizon, a holdout kept in the old state for months gets designed separately.