Experiment Log

Running experiments is the part any team picks up within a few months. What trips teams up is what comes after. You change a screen, check the numbers, turn the winner on for everyone, and move straight to the next topic. What you tried, why you tried it, and what the numbers said live only in one person's memory and a scattering of chat messages. When that person moves to another team, even that goes.

So teams dig the same hole repeatedly. Someone proposes rewriting the copy on the signup screen, and most of the room treats it as a new idea. In fact nearly the same change ran eight months ago and came back with no difference. All that survives of it is one chart image posted in a channel at the time. Nobody can find the image, so the team spends another two weeks on the same experiment.

There's a loss in the opposite direction too: running thirty experiments in a year and having nothing left but the screens currently switched on. A live screen only tells you what worked. Since most experiments end in a loss or no difference, far more information sits in the ones that were thrown away. The log exists to stop both losses at once. It turns the output of an experiment from a changed screen into a conclusion written in sentences.

An entry carries at least seven things. The bar is that someone reading it cold six months later can reconstruct what happened.

1. Title and dates — one line identifying what the experiment touched, plus a start and end date. That one line is usually what a future search matches on.
2. Hypothesis — what you expected to change, which metric you expected to move, and why. A growth hypothesis packs those three into a single sentence, and the "why" is the part that pays off later: when the result contradicts you, that's where you find out what was wrong.
3. What changed — what you actually changed and how, with a screenshot. The person reading six months from now has never seen that screen, so a summary like "improved the copy" carries nothing.
4. Split and size — how users were divided, and how many landed in the control group and the treatment group. The control group is the set left untouched for comparison. Recording the sample size you calculated before starting lets a later reader judge whether enough people actually showed up.
5. Result numbers — both sides of the primary metric and the gap between them. If conversion moved from 24.1 percent to 25.3 percent, write both values and add the statistical significance call, the standard for deciding whether a gap is too large to shrug off as chance.
6. Verdict — pinned to one of three: win, loss, or no difference. Phrasing like "positive signals on some metrics" does nothing when someone is scanning a list.
7. Interpretation and next action — why you think the result came out that way, and what you decided to do about it. The field left blank most often, and the field the next person reads first.

Thirty entries in, people stop reading documents and start scanning the list. For an entry to be filterable at a glance, the verdict has to be one of three values rather than free text.

The person who ran the experiment writes it, during the week it ends. Two weeks later the reasoning behind the choices has already blurred. Keep the length to a single screen. Demand a ten-page report and the log doesn't get written, and what does get written doesn't get read.

Three kinds of people read it.

1. Someone bringing the same proposal again. What they need is to reach the earlier result within 30 seconds. Scanning titles and verdicts to find the relevant entry is enough.
2. Someone designing a similar experiment. Seeing how many users the previous A/B test collected and how many days it took makes this one's timeline far more accurate. An A/B test shows two versions to randomly divided groups and compares the results.
3. Someone deciding what to run next. Reordering the experiment backlog leans on past entries. If three similar attempts in a row came back as no difference, the candidates in that direction all move down.

All three arrive through search. Scatter each experiment into its own separate document and the third reader never arrives at all.

"We tried that last year. Let me find the entry and send you the link."

The proposal isn't new, and the earlier result should be read before the discussion starts.

"The result came out ambiguous, but write it up anyway."

No difference is a result too, so get it on the list.

"There's a verdict but no interpretation, so I can't tell why it lost."

An entry with field seven blank makes the next person run the same experiment again.

At Pagedeck, Mia asks for the write-up on the Friday of the week an experiment ends. It's a recurring calendar block rather than a request, and the rule is that it gets written there, in under 30 minutes. At first only the wins came in. The verdict field hardened into three values after a meeting where the same proposal surfaced for the third time.

Now, when anyone brings a proposal, Mia searches the list first. If something is there, they read it together, and she has the person write one line on what this attempt does differently. Running the same experiment twice isn't wrong in itself. What differs is whether that choice was made knowingly.

Logging only the wins is the first mistake. Treat the log as a record of accomplishments, something to carry into a quarterly review, and there's no visible reason to post an experiment that ended in no difference. But seven or eight out of ten experiments end in a loss or no difference, so keeping only the wins keeps 20 percent of what actually happened. A list built that way does nothing to stop the next person from digging the same hole. The losing entries are what stops them.

Planning to write everything up later is the second. The urgent thing is getting the next experiment live, and last month's numbers are still sitting in a dashboard, so they feel available whenever you want them. What actually survives is the numbers, with the context gone: why that hypothesis was chosen, what went sideways mid-run and how it was patched. Sit down at quarter end to catch up and three or four of twelve entries come back properly, while the rest stay shells holding a title and a verdict.