A/B Test

What shipped used to be decided in a meeting room, and the loudest voice usually won. A team would spend an hour arguing whether the signup button should read "Start for free" or "Build something now," and the call went to whoever held the most senior title. Nobody ever found out whether that call was right. If signups rose afterward, the copy got the credit. If they fell, the season got the blame.

Comparing before and after was the other habit, and it has a hole in it. The copy was not the only thing that changed between last month and this month. The ad budget may have gone up, a competitor may have raised prices, school may have let out. You are left holding the fact that signups rose 8 percent with no way to say where those 8 points came from.

An A/B test closes that hole by moving the axis of comparison from time to people. Visitors arriving at the same moment get split in half: one half sees the old copy, the other sees the new one. Both halves live through identical outside conditions, so whatever difference remains at the end was produced by the copy and nothing else.

One test runs in six steps. The first three finish before anything on screen gets split.

1. Write down in one sentence what changes and which number should move. This pairs one change with one metric. Skip it and you end up scanning for whichever number went up after the fact and calling that a win.
2. Set the minimum detectable effect (MDE). This is the gap you decide in advance is large enough to justify the change. If the team agrees it would switch for a signup rate moving from 4 percent to 4.4 percent, the MDE is 0.4 percentage points.
3. Compute the sample size and convert it into days. The smaller the difference you want to catch, the more steeply the count climbs. If each group needs 8,000 people and 1,000 visitors arrive a day, half to each side makes this a 16-day test.
4. Split the groups by randomization. The side that keeps the old screen is the control group; the side that sees the change is the treatment group. Signup order and the last digit of an account ID are bad dividers, because other properties ride along with them.
5. Leave it alone until the date you set. Watching the running numbers and stopping the moment one side pulls ahead is the peeking problem. With small samples the two numbers swing hard, and if you keep looking, one side is guaranteed to lead at some point.
6. Call it and record it. Statistical significance asks how likely a gap this size is to appear by chance, and the usual bar is that under a 5 percent chance counts as hard to explain by luck. Write down the hypothesis and the result whether the test won or lost.

Change one thing at a time. Alter the button color and the copy together and a difference tells you nothing about which of the two caused it.

Three conditions have to hold at once: enough people arrive, the change is small, and the outcome shows up within days. Signup copy, pricing table layout, notification email subject lines — places where thousands pass through daily and the result is visible on the spot.

The limits mostly come from sample size. A team serving 40 enterprise accounts that splits its contract screen in half puts 20 accounts on each side, and at that size even a doubled difference cannot be told apart from chance.

Changes with slow effects are hard to handle this way too. Shift the pricing policy and the impact lands on churn over six months, and you cannot keep two groups separated that long. Which market to go after is not a question you can put into two versions either.

The method also tells you which side did better and never why. If the reason matters, you have to go ask users separately.

"Let's just A/B this. Two weeks and we'll have an answer."

A proposal to settle the argument with a test instead of opinions. Whether two weeks is actually enough depends on the sample math.

"We don't have the traffic, so we're shipping this one without a test."

Too few visitors to separate a difference, so the call gets made by judgment instead.

"It came out significant, but the gap is 0.2 percentage points."

The result is hard to write off as chance, and the speaker is asking whether a gap that small is worth changing the screen for.

At Pagedeck this test ran on the pricing page copy. Ravi brought two versions, "Start now" and "Try it free for 14 days," and Mia pinned the metric to signup completion rate alone and set the minimum detectable effect at 0.5 percentage points. The math came out to 18 days, and Theo put the assignment code in to split traffic evenly.

Eighteen days later the result was a tie. The difference in completion rate between the two versions sat inside the range chance could produce. Mia wrote that into the experiment log exactly as it came out, so that when someone floats the same idea eight months from now they can see in seconds that it was tried and made no difference.

It's easy to assume that running a test means one side wins. The name suggests a device for picking a winner. In practice most tests end in a tie, and the share that produce a clear win runs somewhere around one or two in ten depending on the company. Teams that don't know this start suspecting a broken setup every time a tie appears and loosen their thresholds, and wins filtered out that way tend to evaporate the next quarter. A tie is not a failure. It's an answer saying this spot needs no more digging.

Judging from the first few days is just as common. A changed screen stands out to existing users, more of them click it, and the early numbers jump. This is the novelty effect, and it usually settles back down after about two weeks. Declare victory on three days of data and you roll out to everyone a change that gained nothing.