Survey

When the only way to learn something was talking to people, what got heard in those conversations tended to turn into percentages by the time it reached a meeting room. Talk to six people, hear the same complaint from four, and the write-up says most users. Four out of six cannot tell you what share of everyone feels that way, but the room receives the number as a distribution anyway.

The mismatch caught up with Pagedeck. Mia proposed grouping documents into folders on the strength of six interviews, and Grace asked roughly how many people across the whole base were in that situation. Four out of six was not an answer that justified a month of two engineers' time, and the proposal slid out of the quarter's priorities.

A survey fills that gap. Why something happens is answered by talking to people; how widely it happens is answered by putting the same question in front of several hundred. The first is qualitative research and the second is quantitative research, and a survey is the instrument used most often on the second side.

Start by writing down, in one sentence, what you will decide once the answers arrive. Without that sentence the question list keeps growing, and when the responses are in nobody can agree which number to look at.

There are six steps.

1. Write the decision rule first. If more than 40 percent say they have failed to find a document, it gets built this quarter; below that, it waits.
2. Decide who gets asked. Sending to everyone who ever signed up and sending only to accounts past their first 30 days produce different numbers. Gathering people who match a condition is research recruiting; in a survey, how you slice the mailing list does that job.
3. Write the questions. Keep it between five and ten, and ask one thing per question. Put speed and reliability in the same question and a low score won't say which one caused it.
4. Send it to five people first. This is where you find questions that two people read differently — one reads "often" as daily, another as once a week.
5. Send it and record the response rate alongside the results. Mail 1,200 people, get 94 answers, and the response rate is 8 percent.
6. Read the distribution. Free-text answers get collected separately and grouped by what they say.

Not collapsing everything into one value is the trick in step 6. Keep only the 3.8 average of a five-point scale and how people divided disappears. Answers piled at 1 and at 5 also average to 3.8, and in that case nobody is actually at 3.8. What that calls for is not an improvement but working out how the two groups differ. A single headline percentage has the same problem: 62 percent says little until you know who the other 38 are.

It fits when you already know what to ask and only the extent is missing. Sorting which statements from earlier interviews are common and which belonged to one person's circumstances is the standard case. It also fits repeated measurement, like asking for a recommendation score each quarter as a Net Promoter Score (NPS) and watching the trend.

Three situations it doesn't fit.

1. Finding out why. Survey answers only come from the options written in advance. A reason outside the options can't be selected at all, so a customer interview has to ask it.
2. Watching people use something. Where people get stuck doesn't surface by asking. A usability test, where someone is given a task and observed, covers that.
3. Asking whether people will use something later. Stated intent and later behavior part ways often. A feature that 92 percent said they would use landing at 4 percent actual usage is not unusual. Intent is cheap to state and expensive to act on.

Go in knowing the limits too. The first is that only the people who answer answer. The delighted and the angry reply first, while the majority who use the product without thinking about it don't. The second is that questions manufacture answers: ask whether slow performance has bothered someone and they will recall a slowness that wasn't on their mind. The third is that splitting a small sample by segment breaks the numbers. Ninety-four answers divided into new and existing accounts leaves one side in the thirties, and a percentage drawn from thirty people settles nothing.

"Let's randomize the option order per respondent. Right now whatever sits at the top keeps coming out first."

The ranking may be a product of the form design rather than what people prefer.

"Forty of the 94 responses wrote the same thing in the free-text box. It wasn't one of our options."

The options drafted in advance failed to cover the actual problem.

"The sample is 94 and splitting it by plan leaves under 30 each. I'll only talk about the overall number."

Refusing a request to slice the figures more finely.

Mia brought the folder proposal back as a survey. Six questions went to 1,200 accounts past their first 30 days and 94 answered. Sixty-two percent said they had recreated a document because they couldn't find it, and 39 percent said it happened at least once a week. Those two numbers settled that it was not something to postpone.

What changed was the thing to build, and the free-text box is why. Eleven people wrote that they wanted folders. Forty wrote that they couldn't find documents through search because they didn't remember the exact title. Mia decided to build search that reaches sentences inside documents before building folders.

The most common reading is that a number makes the evidence solid. A percentage looks settled, and a decimal place makes it look more so. But the value was produced jointly by the questions and the mailing list. Ask the same people the same thing in different words and the number moves 20 percentage points. Leave that unwritten and nobody in the room can argue with the figure, which hands the decision to whoever wrote the questions. Publishing the question text and the response rate beside the number is what keeps it arguable.

Reading a large response count as representativeness is the other one. More responses sounding more accurate feels like it should hold everywhere. But if 3,000 people answered and every one of them uses the product daily, the circumstances of people who left in their first week are not in there at all. Miss that and only the requests of the already-satisfied reach the next quarter's list, while the reasons people leave stay blank.