Sean Ellis Test

There used to be no instrument for deciding whether demand had been confirmed. Whether a product has met the demand of a market is the judgment that sets a company's next quarter, and the evidence behind it was usually a feeling — user numbers going up felt like a yes, visible churn felt like a not yet.

Investor conversations and strategy meetings kept stalling on the same question. Asked whether demand had been confirmed, someone would answer that users love the product, and neither side in the room had any way to test that sentence. The retention curve, which plots how many of the people who signed up in a given week are still around as weeks pass, is the more trustworthy view because it watches behavior. It also takes months to produce an answer, which is too late for a meeting that has to decide now.

The Sean Ellis test exists to get that judgment in a few days. Proposed by the marketer Sean Ellis from data pooled across many companies, it asks users how they would feel if the product went away and reads the strength of demand from the spread of the answers. What defines the question is that it asks about loss rather than satisfaction. People rate almost anything they chose as fine; far fewer will say its disappearance would set them back.

Take one question with three possible answers and compute the share who picked the first. It runs in five steps.

1. Choose who to ask. Narrow to users who have used the core feature at least twice in the last two weeks. Include people who signed up and never came back and you mix in answers from people who never experienced the product, which drags the number down.
2. Use the question as written. "How would you feel if you could no longer use this product?" with three options: very disappointed, somewhat disappointed, not disappointed. Don't expand it to five options or bolt a satisfaction question onto it. The wording has to stay fixed for this quarter's number to be comparable to last quarter's.
3. Compute the share. Divide the count of very disappointed by the total responses. Forty people answer, 19 pick the first option, and the number is 47 percent.
4. Check the response count. Practice puts the floor somewhere between 40 and 100 responses. With ten answers, one person's choice moves the result by 10 points, so whether you cleared 40 is effectively decided by chance.
5. Collect open answers alongside. Ask what people find most useful about the product and who they think it suits best. The percentage tells you the state; these answers tell you why. What to fix next almost always comes out of the second part, and so does the wording other teams will use to describe the product.

The conventional line is 40 percent. Above 40 percent very disappointed, demand is treated as met; below it, there is still searching to do. The figure comes from observing that companies above and below that line went on to grow differently — it was not derived from anything, and it is not a law. Treating 39 and 41 as opposite verdicts reads more precision into it than it holds.

This is a number worth splitting more than reading whole. A product at 31 percent overall often sits at 58 percent among users on teams of five or more. What that says isn't that the bar hasn't been cleared but where it already has been. Narrowing the target to that group, and concentrating there, is the decision this number produces. The overall figure then rises on its own as that group becomes a larger share of who signs up.

There are companion instruments it needs. This survey only reaches people currently using the product. Those who left never receive it, so a high number does not mean people stay. If very disappointed comes in at 52 percent while week-four retention is 6 percent, that 52 measures the temperature of the dozen or so who stuck around, not the size of a market. In the other direction, when the curve settles at some height and stops falling — curve flattening — while the survey reads low, you may be looking at people staying for lack of a decent alternative.

Quarterly is the common cadence. The same eligibility rule and the same wording have to repeat for a change in the number to mean a change in the product. Loosen the conditions and there is no way to say whether next quarter's higher figure came from a better product or a friendlier sample.

"We ran the Sean Ellis test and came out at 34 percent very disappointed."

Below the conventional line, so demand hasn't been confirmed yet.

"It cleared 40, but on 22 responses. I wouldn't decide anything on this one."

Too few answers for the percentage to carry weight.

"It's 31 overall and 58 for teams of five or more. Let's dig in there."

A group that has cleared the bar has been found, so the target should narrow.

Pagedeck first ran it the quarter free signups passed eight thousand. Mia pushed a survey only to users who had created three or more documents in the previous two weeks, and 61 responses came back at 33 percent very disappointed.

Read whole, that's a no. Split by team size, though, it was 57 percent among people using it with five or more colleagues and 14 percent among solo users. Grace narrowed the quarter's target to team users on the spot, and advertising aimed at solo signups stopped in the same quarter. One number produced both decisions.

The most common mistake is reading a pass at 40 as the end of the question. There's a single threshold, so it looks like pass or fail. But the figure is a proportion among people who stayed, which means it can read high even while most arrivals leave within a month. Move to post-PMF on this number alone, scale up sales and advertising, and the people you bring in leak away while the cost stays. That's why it gets read next to an instrument that watches behavior.

Filing it under satisfaction surveys is the other one. It's a survey and it produces a percentage, so the resemblance is easy. The questions differ, though. The Net Promoter Score (NPS), calculated from how likely someone is to recommend the product, measures willingness to recommend; this test measures whether its absence would hurt. Those two don't overlap. Plenty of products are good enough to recommend and easy enough to replace, and the gap between those two facts is exactly what this test is aimed at.