Experimentation

There was a long stretch when the standard way to credit a change was to set it next to the numbers that came after it. The screen changed on Tuesday, signups rose on Thursday, so the screen did it. The entire argument was that the two dates sat close together.

Meetings in that era tended to run the same way. Someone reports that signups rose 3 percent the week the signup button changed color. Someone else mentions that ad spend doubled that week. A third person points out that the week before had been a holiday. Three things changed inside the same window and only one outcome came out of it, so nobody can work out how much of the 3 percent belongs to which. What's left is a contest of conviction, and those meetings usually end wherever the most senior person in the room was already standing.

Digging harder through records that have already been collected does not get past this. What a record can show is which values moved together. That accounts which created a document on day one are more likely to still be around four weeks later is visible in the data; whether pushing people to do that makes them stay is not. The people who were going to stick around anyway may simply also have made a document on the first day.

Experimentation is the procedure built to go around that dead end. Rather than digging further into the past, it manufactures the future in two versions. Incoming users get split into two groups by something as arbitrary as a coin flip, one group sees the changed screen, and the results of both groups over the same stretch of time get compared. Because the split was random, the two groups come out similar on average — same mix of ages, acquisition channels and habits — and the one difference left between them is the screen. When the results diverge from there, the list of things that could have caused the divergence is down to one.

The procedure only runs if the steps ahead of it are in place. Events have to be recorded when a user clicks something, and what counts as success has to be agreed on before anything starts. Experimentation sits in the box after collecting data, cleaning it, and settling what to count. Begin with a weak box in front of it and a gap between the two groups' numbers is not a gap anyone can act on.

The terms under experimentation fall into three groups: how a single experiment gets put together (design), what wrecks the result after it has been put together (risk), and what happens to an experiment once it ends (operations).

Design covers everything that has to be settled before the test starts, and it has eight slots. First how to split, then what gets compared to what, then how much to collect.

1. Randomization — the procedure for splitting users without tilting the groups. Splitting by signup order or by the last digit of an account ID fails because those criteria drag other properties along with them. Early adopters pile up on the earlier side, and a run of consecutive IDs can be twenty accounts one company created at once. Only something close to a coin flip leaves the two groups resembling each other on average.
2. Control group — the group nothing is changed for. It is the baseline the comparison rests on, and without it a conversion rate of 12 percent cannot be called good or bad. Comparing against last month instead is weak, because seasonality and ad spend moved too.
3. Treatment group — the group that actually sees the change. The rule is to change one thing at a time. Alter a button's color, its wording and its position together and a difference in results will not tell you which of the three produced it.
4. Holdout — a comparison group kept aside for a long stretch. An ordinary control group is released when the test ends; this one leaves a few percent of all users on the old experience for six months or a year. If twenty changes were declared winners over a year, this group is the only way to check whether the product actually improved by the sum of those twenty claims.
5. A/B test — the basic form, two versions run side by side. Most experiments that run in practice are this shape, and the other terms here are largely the parts that make this shape work properly.
6. Multivariate test — a form that compares combinations of several elements at once. Two headlines crossed with two buttons makes four arms, and it shows both which combination performs best and whether the two elements affect each other. The cost is that the audience splits across the arms, so a site without plenty of traffic cannot start one.
7. Sample size — how many people have to be collected before a conclusion is possible. It gets calculated before the test starts, and only becomes useful once translated from a headcount into a duration. Needing 8,000 people per arm at 1,000 visitors a day makes this a 16-day experiment at minimum.
8. Minimum detectable effect (MDE) — the smallest difference this test is capable of catching. MDE is locked together with sample size, so wanting to see smaller differences pushes the headcount up steeply. Retune a test from catching a 5 percent difference to catching a 1 percent one and the people required go up twenty-five-fold, not five-fold. Which is why the question to ask first is whether a difference that small would really make the team change the screen.

Risk covers the traps that ruin a conclusion after the design is finished, and it has three slots.

1. Novelty effect — a short-lived lift that comes from nothing but newness. Change a screen and existing users poke at whatever moved, so the first few days run hot. Two weeks in, the number usually settles back where it started, which means declaring victory on three days of data ships a change to everyone that gains nothing.
2. Peeking problem — the error of watching the running numbers and calling the result the moment one side pulls ahead. With a small sample the two groups swing hard, and if you keep watching, one side is certain to lead at some point. Stop there and a gap that opened by chance gets written down as an effect. The defense is to fix the stopping point before the test begins and judge nothing before it.
3. Sample ratio mismatch — a split meant to be even arriving as something like 52 to 48. It signals that assignment went wrong, so the result gets thrown out rather than interpreted. If the treatment screen was slow in one browser and those users left, the two groups stopped resembling each other long before the numbers were read.

Operations covers what the organization keeps once the test is over, and it has one slot.

An experiment log is the single place where what was tried, why, and how it came out get written down. A log containing only the wins is worthless as a record. Most experiments end in a loss or in no difference at all, and it is the losses that stop the next person from digging the same hole. A year later, when somebody who just joined proposes rewriting the copy on the signup screen, the log's job is to show within thirty seconds that this ran eight months ago and changed nothing.

The three groups run in time order. Design settles what happens before the start, risk trips teams up while the test runs and at the moment of judging, and operations keeps what is left afterward. Plenty of teams do the first two and skip the last, and then, 300 experiments later, the only thing the company has accumulated is whatever screen happens to be live.

"This gets us as far as correlation. If you want to call it a cause, we have to run an experiment."

The records on hand show only that two things moved together, so anyone wanting to say cause has to split the users and try it.

"The treatment is ahead right now, but we still have two more days to run."

The mid-flight numbers look good, and the speaker is refusing to judge before the duration set in advance is up.

"This one came back as no difference, and I logged it that way."

Not a report of failure — a note that this direction has been checked and found ineffective, so nobody has to dig here again.

At Pagedeck the whole procedure went around once on the first screen a new user sees. Mia thought that showing an example document right after signup, instead of an empty screen, would raise the number of people who create their first document. Her evidence was the observation that accounts which created a document on day one were far more likely to still be active four weeks later.

Theo's first question was whether that observation was a cause at all. People who made a document on day one might simply have been the motivated ones, in which case nudging everyone guarantees nothing. So the two of them did the arithmetic first. They set the rule that conversion, currently 34 percent, would have to reach 39 percent for the screen to change, and the calculation said catching a difference that size takes 6,000 people per arm. At 800 signups a day, filling both arms takes about two weeks.

They agreed not to look at the numbers for those two weeks. Ravi passed along on day five that the treatment was ahead, and the judgment was held anyway; by day ten the order had flipped. The final numbers after two weeks were 34 percent and 36 percent. That fell short of the line set in advance, so the screen stayed as it was, and Mia wrote the number into the log along with one sentence — the example-document approach moves this much and no further.

The most common mistake is treating this as a procedure for finding winners. The sessions where results get shared are mostly filled with successes, and the ones that never made it into the room are invisible, so people get the ratio wrong. In practice seven or eight out of every ten experiments lose or come back as no difference. A team that will not accept that number starts handling results differently: losses go unshared, ambiguous results get sliced down to the window where the treatment led, and a flat outcome gets re-cut until some segment looks good. Numbers picked that way do not reproduce next quarter. What experimentation actually hands a team is not victory but a cheap way to abandon a wrong direction. A direction dropped after a two-week test costs roughly a twelfth of the same direction dropped after six months of building.

The second mistake is believing that anything at all can be decided this way. Deciding with data sounds reasonable, and switching a test on inside a modern tool takes a few clicks. But the method comes with a headcount condition attached. On a service with 200 visitors a day, catching a 3 percent difference takes months, and over those months other parts of the product keep changing until the comparison itself falls apart. It also cannot be used for decisions that are hard to reverse. Rewriting the whole pricing structure, or changing the company name, is not something you can show to half the users. A team that misses this limit and defers every decision until after a test spends three months running something that will never conclude while nothing gets decided. A team short on users learns far more in the same two weeks by sitting five people down and watching them use the screen.