Growth Experiment

Decisions about the product used to be settled in a meeting room. What the signup button should say, whether the free trial runs seven days or fourteen, whether the pricing page has three columns or four. There is no correct answer to any of these, everyone arrives with a different picture, and so the meeting runs long while the conclusion drifts toward whoever holds the most senior title.

Pagedeck kept staging the same scene. Noa argues that "Start for free" lowers the barrier. Ravi argues that "14-day free trial" communicates the value more clearly. Forty minutes in, Grace ends it with "let's go with the first one." The screen ships with neither of them convinced, and when signups come in eight percent higher a month later, nobody can say whether that was the wording or the blog post that went out the same week. The next time the wording comes up, none of the previous reasoning survives. Another forty minutes disappear, and this time the room might tilt the other way.

Growth experiments start from the idea that when opinions split, users should answer instead of people. The procedure has three steps. First, write down as a prediction which number a specific change should move and by how much. Second, split arriving users at random into two groups and show one the current screen and the other the changed one. Third, decide whether the gap between the two groups is larger than chance would produce on its own.

Writing the prediction first has a specific purpose. After results are in, any number attracts a plausible story. If it goes up, lowering the barrier obviously worked. If it goes down, the value clearly wasn't communicated. Both stories were assembled after the fact, so neither helps with the next decision. If the third field was supposed to be the problem and removing fields changed nothing, you learn that field count wasn't it, and the next candidate looks different. Running both versions side by side during the same period has a purpose too. Comparing the month before a change to the month after drags every holiday, campaign, and seasonal swing in between into the result. Randomly splitting people who arrived the same day at the same hour makes the two groups identical except for the one screen, which is what lets you narrow the cause to that screen.

The terms under growth experiment fall into four groups. One is where a single attempt gets written down (design), one is how many attempts get queued and turned (operations), one is the gauges for whether that turning is working (metrics), and one is the problem that shows up when the turning goes well (risk).

Design is where an attempt gets pinned into a sentence. A growth hypothesis states what changes, which metric should move and by how much, and why you believe it will. "Cutting the signup form from five fields to two raises signup completion from 24 percent to 30 percent, because the drop-off concentrates at the third field asking for company and job title" is one example. Leaving the reason blank makes the result unreadable, because both the win and the loss only mean something when held up against what you expected.

Operations handles attempts as a queue rather than one at a time, and splits in two: deciding what comes first, and deciding when it runs.

1. Experiment backlog — the list of candidates in priority order. Ideas pile up to about thirty while a quarter has room for five or six, so the list's real job is closing things down rather than holding them. Priority usually multiplies three factors: how far the number would move if it works, how confident you are that it will, and how many days it takes to build. A candidate with a large payoff and six weeks of work loses to three two-day candidates, because three verdicts in the same span beat one. Keeping the closed items visible matters too — the twenty-five you passed on come back as fresh ideas two months later if nobody wrote them down.
2. Growth sprint — the fixed cadence that turns the list. Two weeks is common. Day one reads the last round's results, re-sorts the backlog, picks the two or three to run, then the work gets built and shipped, and the group meets again on the last day. The cadence exists because without a date on the calendar, something more urgent always takes the time. A date means there has to be a result to read, which means something had to go live before it. The rhythm also gives the backlog a fixed moment to be re-sorted, so a candidate that mattered a month ago gets re-scored against what the team has learned since.

Metrics watches whether the turning is healthy, and the two terms only mean something together.

1. Experiment velocity — how many experiments finished in a given period. A team running four a quarter and a team running twenty-four end the year holding very different amounts of improvement even at the same hit rate. When this number is low the cause is rarely a shortage of ideas; it's the time between deciding and shipping. Three approvals and a week of release queue per test caps you at four a quarter no matter how long the backlog is.
2. Win rate — the share of experiments that produced a clear improvement. One to three out of ten is typical. A low figure isn't bad in itself, and an unusually high one is a warning. Winning eight out of ten means you're only running things you already knew the answer to, so almost nothing is being learned. Velocity alone with a hit rate near zero means poking at random; a high hit rate with low velocity means holding one safe bet for too long.

Risk is the group covering what happens when the turning goes well. A local maximum is the state of repeating small improvements while missing a larger jump. Button colors and copy get tuned until conversion climbs from 22 to 26 percent, while a competitor who changed how signup works at all is sitting at 41. Climbing uphill by feel in fog gets you to the top of the hill you're standing on every time, and never to the taller mountain beside it, because measuring one step at a time says nothing about the route there. Teams that mostly run small tests therefore reserve a slot each quarter for a candidate that changes the shape of things. Those candidates score low on confidence and always lose the sort, so without a reserved slot they never come up.

The four groups form one loop. Design turns an attempt into a sentence, operations puts that sentence in order and runs it on a schedule, metrics show how often the loop turns and how often it lands, and risk asks whether the loop is circling in place. One more piece sits outside the loop: reading the results belongs to growth analytics, so the whole procedure falls apart in a product where records aren't kept. Which group a user landed in and what that user then did both have to be stored before any gap can be calculated, and that assignment has to be stable so the same person keeps seeing the same version on every visit.

"This one came back not significant. That doesn't mean no effect, it means we don't know yet."

The gap sits inside the range chance would produce, so don't file it as a loss.

"Write the hypothesis first. Without a number you expected, there's no way to read the result."

An experiment run without a prediction gets its interpretation attached afterward, whatever comes back.

"We're winning six out of ten this quarter. I think we're only picking safe ones."

A high hit rate read as a signal that the team is learning very little.

Pagedeck's growth sprint starts every other Monday morning. Ravi reads out the three results from the last round. Cutting the signup fields moved completion from 24 to 29 percent, so it stays. Adding customer logos to the pricing page produced a 0.4 point gap, so it comes down without a verdict. The third one only drew 900 people into one arm over ten days, so it runs two more weeks before anyone calls it.

Then the backlog gets re-sorted. Mia brings four new candidates, Ravi attaches an expected effect, a confidence, and a build cost to each, and Theo says which of them can ship within two days. Usually half get closed right there. Two or three are picked, and the rest stay on the list knowing they may lose the same sort next round. For each one picked, the group writes down how long it runs and how many people have to land in each arm before anyone calls it. Decide that later and you end up moving the line after seeing the result.

The most common mistake is treating a losing experiment as waste. Two weeks of work gets deleted because the result was bad, and the time looks thrown away. But the output of an experiment is the verdict, not the screen. If half the team believed a fourth pricing column would lift paid conversion, one result saying otherwise saves the six weeks that belief was about to cost next quarter. Teams that don't count that value drift one of two ways: they run only the tests they expect to win, pushing the hit rate to eighty percent while learning nothing, or they refuse to delete losers and fill the product with screens that contradict each other. Starting from the assumption that more than half of what you build gets deleted avoids both. Making things easy to remove is worth as much here as making them polished.

Calling a lead a win is just as common. On day three the changed arm is up three points, and the urge to stop and announce it is strong — partly because it really is ahead. But while the sample is small, two arms trade the lead constantly for no reason at all. Flipping a coin twenty times and getting thirteen heads is not unusual. A team that stops whenever the result looks good keeps recording differences that aren't there, and six months later the sum of all those improvements is nowhere to be found in the actual metric. So the run length and the number of people required in each arm get fixed before the test starts, and nobody opens the dashboard until the test reaches that line.