Validated Learning

Measuring progress by what got built felt natural for a long time. Seven features shipped this quarter, all on schedule, is a good quarter.

One Pagedeck quarter went exactly that way. All seven shipped and nothing slipped. The share of free teams converting to paid was identical at the start and the end. In the retrospective Mia asked not what had gone unfinished but what the team had learned, and nobody could answer.

It isn't that nothing was learned building seven features. The learning just stayed in people's heads as impressions. One person felt templates were working and another felt they weren't, and there was no basis for telling which was right. So the next quarter's plan got built the same way, on the louder impression rather than the better-supported one.

Validated learning shifts the unit of progress from output to learning, with a condition attached: the learning has to be confirmed rather than felt. A quarter gets measured by how many hypotheses reached a verdict, not by how many features shipped. That change makes some productive-looking quarters score badly, which is the point of it.

Three conditions have to hold for something to count as learning. A hypothesis written before starting, data from actual user behavior, and a conclusion capable of changing the next decision. Miss any one of the three and what you have is interpretation rather than learning, however confident it sounds in a review.

The procedure has five steps.

1. Write the hypothesis as a falsifiable sentence. A hypothesis is a guess written so it can be shown true or false. "Templates will help" can't be checked; "at least 30 percent of free-team admins use a template at least once in their first week" can.
2. Write the threshold and the window in advance. Get that order wrong and you end up fitting the threshold to the result. Settle on 20 percent after seeing 22 and every experiment succeeds.
3. Choose the cheapest way to check. Often the feature doesn't have to be finished. Ship an MVP, split two versions with an A/B test, or listen first through a customer interview.
4. Measure and rule. The result is one of three: true, false, or no verdict. No verdict means the sample was too small or the threshold was vague, and no conclusion gets forced onto it.
5. Record the conclusion as a sentence usable by the next decision. Not "template usage is low" but "first-week template usage is 9 percent, and teams that used one convert to paid at 2.4 times the rate of teams that didn't."

A false hypothesis is learning too, and often the more valuable kind. Expecting 30 percent and finding 9 rescues a quarter that would have gone into building more templates. Learning doesn't require the experiment to succeed; it requires a verdict. Teams that only count successes end up designing experiments they already know the answer to.

It fits where uncertainty is high: a new product whose users are unknown, a market nobody has entered, the period before committing a large investment. It's also the basis for deciding a pivot. A pivot is changing direction on the strength of what was learned, and without adjudicated hypotheses changing direction is just changing your story.

There are places it doesn't fit.

1. Building something already confirmed. A feature an enterprise customer wrote into a contract isn't a hypothesis to test; it's work to do, and framing it as an experiment wastes everyone's time.
2. Work fixed by regulation or law. Whether to do it isn't settled by data, only how to do it well.
3. Stages without enough users to reach a verdict. At thirty visits a day no sample will separate two conversion rates. Interviews replace numbers there, and the conclusions stay qualitative until the traffic arrives.

Putting it into practice means changing the shape of quarterly goals: a list of hypotheses to adjudicate instead of a list of features. Write five and check at quarter end how many got a verdict. Take care not to use a vanity metric as the threshold — a vanity metric looks good and is useless for decisions, and building one like cumulative signups into the criteria means no hypothesis can ever turn out false.

"I wrote this quarter's goals as five hypotheses instead of five features."

Move the unit of progress from output to verdicts.

"Let's fix the threshold before we start. Decide after the result and everything succeeds."

Prevent the threshold from being fitted to the outcome.

"Nine percent isn't a failure, it's a finding. We nearly spent another quarter on templates."

A false hypothesis carries value.

Pagedeck ran this for one quarter and then reshaped it. Mia wrote eight hypotheses the first time and two reached a verdict by quarter end; the other six never got as far as a designed experiment.

So the count dropped to three, each with its checking method and window written alongside. A quarter producing three verdicts settled more than a quarter producing two, and fewer hypotheses turned out to mean more of them getting answered. Grace started using the list in investor updates as well, because what was learned reads as a basis for the next quarter's plan in a way a feature list doesn't.

The most common reading is that having data makes it validated learning. A conclusion drawn from numbers looks validated. But a conclusion attached after looking at the data, with no hypothesis first, is interpretation. Open a dashboard, pick the number that stands out, and build a story — any conclusion becomes available, including the opposite one. Reverse the order and the same data supports whichever claim you want. A sentence written first is what lets data deliver a verdict. Without it the data is evidence for whoever reaches it first.

Reading it as permission to fail is also frequent. Saying a false hypothesis has value sounds like an exemption. But the value appears when a verdict arrives and that verdict changes the next decision. Checking the same thing three times, or reaching a verdict and leaving the plan untouched, is time rather than learning. Recording what each verdict stopped and what it started, alongside the count of experiments, makes that distinction visible. A quarter of verdicts that changed nothing is a quarter spent measuring.