Sample

The thing you want to know is the whole, and there has always been a gap between that and what can be looked at. Learning how ten thousand users feel about a new screen would mean asking ten thousand people, who won't answer, and whose answers couldn't be read if they did. A usability session takes an hour per person, which makes ten thousand an impossible number. The gap is structural rather than a matter of effort.

So conclusions started coming from a part of the whole. The problem is that the part can differ from the whole. There was a meeting that concluded from five people that everyone wanted a feature, and all five had been invited in by the same team. Nothing established that what that team wanted matched what ten thousand wanted.

What surfaced there is that the value of a conclusion is set by how the part was drawn, not by the fact that a part was drawn. Sample is the name given to that drawn subset, and the name exists separately to force the method and the size to be written down every time. Those two lines are what a later reader needs in order to trust anything.

Define the whole first, then the method and the size. Five things to watch.

1. Write the whole as a sentence. The whole you want to know about is the population, and "our users" is not one. "Users on paid accounts who logged in at least once in the last 30 days" is. Without that sentence, who the conclusion applies to is also undefined. Most arguments about a result turn out to be arguments about this sentence.
2. Decide how to draw. The standard is at random: everyone in the population needs an equal chance of being picked for the subset to represent it. Drawing from whoever is within reach introduces a skew in the selection process, which is selection bias. Asking five colleagues in the office is the most common instance.
3. Decide the size. It varies enormously with what you're measuring. Stating a proportion as a number takes hundreds to thousands of people; finding where users get stuck takes five. Try to do both with the same five and only the second one holds. Deciding which of the two you are doing comes before deciding the number.
4. Attach a width to the result. A value from a sample is a range, not a point. The range the true value is believed to sit in is a confidence interval, and 40 of 100 answering yes gets written as somewhere between 30 and 50 percent rather than as 40. Drop the width and next month's 34 percent reads as a decline.
5. Count who didn't answer. Send a survey to 5,000 and get 200 responses and the sample is 200, not 5,000. And those 200 are the people interested enough to answer, which makes them likely to lean away from the whole in some direction. Reporting the response rate alongside the result is the cheapest correction available.

The purposes split in two, and the required size splits with them: measuring a number, and finding a problem.

Measuring needs a large sample. A survey, asking many people the same question to see the distribution, belongs here, as does an A/B test, which splits traffic at random between two versions and compares outcomes. With a small sample in an A/B test the difference between the two versions can't be told apart from chance, so the experiment answers nothing at all. On a screen converting at 3 percent, seeing a meaningful difference takes thousands per side. Running it with fewer costs the same time and produces nothing.

Finding problems works with a small sample. A usability test, handing someone a real task and watching where they stall, surfaces most of the large problems with five people. Three stalling in the same place is a problem to fix, and adding twenty more people mostly means seeing that problem twenty more times. But those five can't produce a proportion. Three of five is not 60 percent, and writing that figure into a report hands the reader a confidence nothing supports. The list of stalls is the honest output of five sessions.

Log data is usually a census, which is worth knowing. Click records and order records aren't a drawn subset; all of it is there, so there's no sample to define. The exception is analytics tools that compute from a fraction when the volume is large, which makes checking whether a displayed number is complete or partial a habit worth keeping.

"Three of five got stuck. It's a real problem to fix, but don't write 60 percent."

Separating what this sample can say from what it can't.

"5,000 sent, 200 answered. The sample is 200."

Counting without the response rate isn't allowed.

"Everyone in this group came in through an invite."

There's a skew in how they were drawn.

Pagedeck keeps the two purposes apart. When Noa runs usability sessions it's five people, and the output is written only as a list of places people stalled. When Mia wants satisfaction as a number, it goes to 400 randomly drawn accounts and the result carries a width.

The absence of that separation caused a problem once. Four of five people in a usability session called the new screen good, and that result nearly went into investor material as "80 percent satisfied." Mia cut the line on the spot, and since then the sample size and the drawing method get written next to every result. Two extra lines removed the whole class of mistake.

The most common belief is that a large sample is an accurate one. A big number looks trustworthy, and it's true that the width narrows as the count grows. But the count comes after the method. Gather 50,000 through a skewed method and you get a wrong conclusion with 50,000 behind it, and the narrowed width makes that conclusion look more certain. A survey link posted to one community and answered by 30,000 people is the standard example. Count improves accuracy only once the condition of drawing at random has been met.

The opposite belief also comes up often: that a small sample supports no statements at all. Five people can't produce a proportion, and they can establish that a problem exists. If three couldn't find the same button, that button is hard to find, and that conclusion needs no sample size. Teams that shelve a result for being a small sample spend weeks rediscovering what they already knew. Sample size determines what can be said, not whether anything can be. Knowing which of the two you have is most of the skill.