Quantitative Research

Teams used to take what users said and use it directly as evidence. If three of five people in interviews said search was frustrating, that became "users find search frustrating" in the next meeting. The sentence carried weight in the room, and it was missing the one thing a decision needed.

What was missing was size. To choose which of two requests gets built first, you need to know how many people each one touches. Yet "customers keep asking for this" met with "how many customers?" produced no answer at all. Three out of five and twelve thousand out of twenty thousand call for entirely different decisions, and the first kind of research cannot produce the second kind of sentence.

Quantitative research exists as a separate branch to fill that gap. User research splits in two by what it returns: the side that finds out why people behave as they do, through their words and through watching them, is qualitative, and the side that measures how common that behavior is, is this one. They aren't options to pick between — they run in an order, and most arguments about which one is better are really arguments about which one comes first.

You ask the same question of enough people, or count behavior that has already been recorded, and turn the answer into a proportion. It runs in five steps.

1. Rewrite the question so a number can answer it. "Is search frustrating?" cannot be counted. "What share of users who searched in the last 30 days closed the window without clicking a single result?" can.
2. Choose how to measure. Asking means a survey, the same set of questions put to many people. Watching means event tracking, which records what people do inside the product, and funnel analysis when the question is how many people drop out at each step. Where words and behavior disagree, behavior is the one to trust, since nobody has to remember anything for it to be recorded.
3. Work out how many responses you need. The minimum number of observations required before a conclusion holds is the sample size. Ask ten people and write down 60 percent and you have written down six people, a figure that swings to 50 or 70 when one person changes their mind.
4. Collect and count. What usually breaks here is who got collected. Run a survey only inside the product and nobody who already left can answer, so the opinion of the people who stayed looks like the opinion of everyone.
5. Check whether the gap is chance. The standard for deciding that a difference is hard to explain by chance is statistical significance. Split 200 people into two groups, get 52 percent and 48 percent, and that gap is not yet large enough to say anything about.

There is one thing this kind of research cannot answer. It gives you the proportion and withholds the reason. A 34 percent drop-off after search says nothing about why those people left, and that answer only comes from sitting with them.

It goes wherever a proportion decides something. Four of those places come up repeatedly.

The first is setting build order. If one request affects 18 percent of all users and the other affects 0.4 percent, the argument about sequence generally ends there.

The second is judging whether a change was an improvement. An A/B test — showing two versions to randomly split groups and comparing the outcomes — is one form of quantitative research. The condition is deciding the criteria in advance and waiting until enough people have passed through, because a test read early reads as whatever the first few hundred visitors happened to do.

The third is finding where people stop. If there are four steps between signing up and writing a first document and half the group falls out at step two, this is the research that points at step two.

The fourth is tracking a state over time. The Net Promoter Score (NPS), calculated from asking how likely someone is to recommend the product, and monthly active users (MAU), the count of people who used it in a month, are numbers measured the same way every month so the trend can be read.

The places it works badly are just as clear. Reactions to a feature that doesn't exist yet, or an account of what frustrates someone, cannot be obtained this way. For those, qualitative research finds out what is happening and produces a hypothesis, and the quantitative pass then measures how often it happens. Reverse the order and you stall on what to even put in the survey.

"What percentage of everyone is that?"

A request to check the size of a problem that arrived as an anecdote, before it gets planned around.

"The sample is 40 people, so I wouldn't read anything into that gap yet."

The difference could be chance, so it shouldn't be used as a conclusion.

"The survey only gets us the proportion. The why has to come from interviews."

The question has moved past what this method can answer.

At Pagedeck the quarter that search got fixed ran in exactly this order. Mia met eight users and heard that they didn't trust search results and dug through folders by hand instead, and the next job was measuring how common that was.

Counting the records showed 34 percent of users who searched were closing the window without clicking any result. What eight people described turned out to be a problem affecting somewhere over six thousand of twenty thousand users, which moved search out of the pile of things that keep getting deferred.

The most common mistake is believing the number carries the reason with it. The figure arrives with such precision that the explanation feels like it came too. But 34 percent reports what is happening and stays silent on why. Try to read the reason out of the number and what usually gets substituted is the builder's own guess, and the feature fixed on that guess ships with the number unchanged.

Treating a large response count as trustworthiness is the other one. A survey with 3,000 responses looks sturdier than one with 40. What actually wobbles, though, is the source more than the count. If all 3,000 are people still using the product, not one departed user's reason is in there. Lean on that survey while churn is the problem and you will read a satisfied result while the people leaving stay invisible.