Experiment Velocity

Once the idea spread that growth comes from many small attempts rather than one large redesign, teams got serious about judging whether an experiment worked. Did this version beat that one, and is the gap more than noise? Yet teams kept showing up six months later with flat numbers and no idea what they were doing wrong, because every experiment they ran had been read correctly.

The missing question was how many. A team that runs four experiments a quarter and a team that runs twenty-four can win two out of ten at exactly the same rate and still end the year six times apart — three improvements banked against nearly twenty. When someone asked in a review whether the team was doing experiments well, the conversation was all wins and losses. Nobody counted attempts, so attempts never entered the answer.

Experiment velocity fills that gap. It ignores how often the team was right and looks only at how often the team went. Start counting it and, for the first time, a flat quarter splits into two different diagnoses: the ideas were bad, or there simply weren't many attempts.

Count the experiments that finished inside the chosen period — read to a result, with a decision made to keep or drop the change. Three steps.

1. Pick the period. A quarter is common; teams running experiments constantly look at it monthly. Too short a window and a single experiment swings the number enough to hide the trend.
2. Define what counts as one. Count finished, not started. Finished means the planned duration or sample was reached, the result was read, and somebody decided what to do about it. An experiment left switched on with nobody opening the results doesn't count.
3. Count them. Twelve finished in Q3 makes the velocity twelve per quarter, or four a month.

Two cases sit on the boundary. One is the experiment killed mid-flight — switched off during an outage, or invalidated by a setup mistake. Leave those out; they're accidents rather than attempts. The other is the experiment that lost, which always counts. Losing is a result that changes the next candidate, and dropping losses mixes this number together with win rate until neither one can be read.

With several teams, some divide by headcount. Eight people finishing twelve a quarter against three people finishing nine puts the second team twice as high per person. The division isn't there to rank teams, though — it's there to test whether adding people raises the count. Usually it doesn't, and that by itself says the bottleneck isn't staffing.

There is no absolute number that counts as enough. What sets the ceiling is traffic, not willpower. Separating a 24 percent signup completion rate from a 30 percent one takes roughly a thousand visitors in each branch, and that requirement — the number of visitors one conclusion costs — is the sample size. At 400 visitors a week, one experiment takes five weeks and the quarter caps out at two or three. Move the same team onto a product with 100,000 visitors and it can finish one a day.

So the benchmark is last quarter's own number, not another team's. Four per quarter rising to nine is progress; twelve falling to seven is a question about what got slower.

When the number is low, the cause is rarely the ideas. It's far more common to have thirty candidates sitting in the experiment backlog and still manage only four a quarter. The jam is almost always in the path to launch. Three approvals, a week of deploy queue, and three days of waiting to see results will hold the count down even with a hundred candidates waiting.

Read alone, the number also misleads. Twenty-four a quarter with a win rate near zero means the team is poking at random; a 50 percent win rate at three a quarter means it only touches sure things. The two numbers mean something only side by side.

"We finished twelve experiments last quarter and seven this one."

The count dropped, so it's time to find where things jammed.

"It's not that we're short on ideas — it takes three weeks to get one live."

A diagnosis that locates the problem in the process, not in the candidates.

"Let's take the hit on velocity this quarter and run one big thing properly."

A deliberate decision to push the number down.

Pagedeck runs a growth sprint every two weeks and records how many experiments closed in each round. At the end of the quarter Ravi reports the total to Grace, the CEO. When twelve became seven, what Grace asked was not why the team fell short but where the work had been sitting.

The answer that came back was security review. Three experiments touching the billing screen had each waited four weeks for sign-off, and nothing else started in the meantime. What the team changed the next quarter was the order — candidates that stay away from billing go first — and the count came back to eleven.

The most common mistake is reading velocity as a target to raise. A rising number looks like a faster team, which makes it tempting as a quarterly goal. But the number never asks what was run, so the easiest way to lift it is to run only small things. Button colors and copy tweaks can close two a day, and a team handed that target arrives six months later at thirty a quarter having never once touched anything as large as the signup flow. That is how a local maximum gets built — endless small gains while the bigger jump goes untried.

Treating it as a synonym for engineering speed is the other frequent slip, since both sound like shipping faster. But the denominator here holds more than build time: it also holds the wait for results and the time it takes someone to read them and decide. An experiment built in a day, filling its sample over two weeks, then sitting five days for interpretation barely moves the quarterly count even if engineering doubles its pace. Teams serious about this number start with approval steps and analyst queues rather than with code.