Win Rate

The first problem a team hits after making experimentation routine is the unit of reporting. Usually the only number available for the quarterly meeting is a count: we ran 40 experiments. That communicates effort. It says nothing about what got better.

The scene played out at Pagedeck. Ravi reported the 40, Grace asked how many of them won, and the answer was that nobody had counted. Each experiment's result sat in a document; the set had never been looked at as a set. Counted up, 9 had moved a metric.

The number matters because a count alone can't separate two very different states. A team that ran 40 and won 9 and a team that ran 40 and won 2 did the same amount of work with very different judgment about what to test. The second team should fix how it chooses before it tries to run more, because doubling the count of badly aimed experiments doubles the cost and not the result.

Divide experiments won by experiments finished. Three steps.

1. Fix what's counted. Only experiments that ended inside the window go in. Include the ones still running and the rate inflates, because experiments with clear positive results tend to finish first.
2. Write the success criterion before starting. Which metric has to move, by how much, and where the line sits for calling the difference unlikely to be chance. That last line is what statistical significance covers, and making the judgment at all requires the sample size — the minimum number of observations needed to conclude anything — to have been reached.
3. Divide. Nine of 40 finished experiments past the bar is 22.5 percent.

Step 2 is the hard one in practice. Decide the criterion after seeing the results and any experiment can be made to look like a win: click-through didn't move but time on page did, or there was no overall difference but new users improved. So the hypothesis — what's being tested and what counts as success, written down as something checkable — gets recorded before the experiment starts, and the win rate is counted from that record only.

Finished experiments fall into three groups: won, lost, and undecidable. The last is where the sample was too small to say whether there was a difference at all. Whether those belong in the denominator varies by team, and they do belong. Designing an experiment in a place where the sample can never accumulate is also a choice about what to test, and excluding it makes that mistake vanish from the number. Pagedeck's 40 broke down as 9 won, 19 lost, 12 undecidable. Drop the undecidable ones and the rate climbs to 32 percent, while what the quarter actually produced is still 9 wins.

The band most often quoted runs from 10 to 30 percent. Winning once or twice in ten is normal. If that sounds low, it's worth revisiting what an experiment is: a change whose outcome you already know is just an improvement, and testing things whose outcome you don't know means being wrong most of the time.

So a high number isn't a good number. A team at 80 percent doesn't have better judgment — it only tests things that were going to win. Twenty experiments on button copy mostly win and mostly move nothing. A team at 20 percent, meanwhile, can lift signup conversion from 24 percent to 41 percent with one of its winners.

This is why the figure never travels alone. Multiply it by the number of experiments run in the period to get the wins the quarter produced: 22.5 percent of 40 is 9, while 40 percent of 10 is 4. The first team got more. Add how far each winner moved its metric and the picture is complete. A quarter of two wins that each shifted a headline number beats a quarter of ten wins worth a tenth of a point apiece, and only the third figure tells you which quarter you had.

The direction it drifts over time is worth knowing too. A team new to experimentation starts low, because it doesn't yet know what to test. Once it starts locating where users actually fall out — a funnel leak — and aiming there, the rate climbs. After a while it falls again, because the easy wins in that spot are used up. That decline is not a warning sign. Read alongside where the team is testing, it usually means the team has stopped harvesting a place it already understood.

"Win rate this quarter was 22.5 percent. Nine of 40 experiments won."

Reporting the work done and the value obtained in one line.

"Don't set a target for win rate. We'll end up only testing what we know wins."

The easiest way to raise this number is to narrow what gets tested.

"The 12 undecidable ones stay in the denominator. Designing for a reachable sample is part of the skill."

Blocking a calculation that would flatter the number.

Pagedeck's last three quarters came in at 11 percent, 22.5 percent, and 16 percent. The middle quarter was high because the team aimed only at the step losing the most users, and the third dropped because the easy wins in that step were gone and the work moved to unfamiliar ground.

Mia didn't raise that decline as a problem in the quarterly meeting. She put the total metric movement from the winners beside it instead: the quarter at 16 percent had lifted signup conversion more than the quarter at 22.5 percent. Fewer wins, each one larger.

The most common mistake is setting this as a target. A number written as a percentage looks easy to aim at. But the fastest way to raise it is to pick only experiments that were going to win. A team handed that target fills the backlog with button colors and copy tweaks, and six months later the win rate reads 45 percent while the metrics sit where they were. This is an instrument; the thing to raise is on the metric side.

Reading lost experiments as waste is the second mistake. Time went in and nothing visible came out. What a loss actually leaves is the information that this direction doesn't work. Try adding testimonials to the signup screen, see no difference, and the reason to touch that spot with testimonials again is gone. Without a record, another person runs the same experiment six months later. So losses get their reasoning written down as well, and counting them is exactly what the denominator of this metric is for.