Selection Bias

Sometimes every calculation is correct and the conclusion is still wrong. Nothing was counted incorrectly; the wrong people were counted.

Ravi once ran a satisfaction survey at Pagedeck — the same questions put to many people to see the distribution — and 412 responses came back averaging 4.6 out of 5. He reported that users were happy with the product.

The catch was that the survey link went out as a banner inside the app. Anyone not opening the app never saw the banner, and anyone who had already left certainly didn't. The 4.6 was the score of people currently using the product. Not one of the thirty accounts that churned that month appeared in it. The average was arithmetically right; the population behind it wasn't the one anyone thought they were reading about.

Selection bias is the name for that condition, and it's a different problem from having too little data. Too little data can be fixed by collecting more. Data collected through a skewed path only skews further in the same direction as it grows.

It appears whenever the way a sample gets chosen is entangled with the thing being measured. If only people who open the app can answer, and opening the app is related to satisfaction, the sample tilts toward the satisfied.

There are three places it gets in.

1. Who was asked. When the recruiting path reaches one group, that group's characteristics become the result. An in-app banner, a newsletter list, a post in one community are all paths of this kind.
2. Who answered. Even asked the same question, respondents select themselves. A satisfaction survey draws the delighted and the furious while the middle stays silent. If the response rate is 3 percent, the question is what kind of people that 3 percent are — which makes response rate a number to report rather than a footnote.
3. Who was kept in the analysis. Filtering out values that look wrong belongs here. Decide the rule after seeing the results and the rule drifts toward the conclusion you were hoping for. Write the filter conditions down before opening the data.

That it doesn't go away with a bigger sample is the defining property. The sample size — the minimum number of observations a conclusion needs — is about reducing chance, not about removing skew. Take 412 responses to 4,000 and they're still the scores of people who open the app.

The way to check is to compare the sample's composition against what you already know. If 70 percent of your customers are on individual plans and only 20 percent of respondents are, the sample tilted toward teams. Comparing proportions on values you already hold — plan, signup period, company size — usually reveals which direction the tilt runs.

Three things reduce it. Spread recruiting across several paths so no single one carries the result; compare respondents against non-respondents on the values you know, to see how they differ; and state whatever skew remains in the conclusion itself. That last one turns "satisfaction is 4.6" into "satisfaction among users who opened the app in the last 30 days is 4.6." None of the three makes the skew disappear; together they make its direction known.

The most frequent setting is surveys and interviews. How you recruit participants is how you choose your sample, so where the notice was posted decides half the result. Offer an incentive and you gather people who respond to incentives; post in a community and you gather that community.

The second is experiments. When two options are compared, who lands in which group has to be assigned at random or the results can't be set against each other. Show a new screen only to people who opted in and compare it with the old one, and what's being compared is the enthusiastic against everyone else. That gap wasn't produced by the screen. Random assignment is what buys the comparison, and it's cheap to get right up front and expensive to retrofit afterwards.

The third is customer stories and advocacy metrics. Interview only satisfied customers and assemble the answers into an explanation of why the product sells, and the explanation already contains its own conclusion.

The fourth is assembling training data. Collect examples from one group only and the skew carries into the system's output, which is handled under its own name as AI bias — skew in training data turning into discriminatory results.

"That survey only ran inside the app. Nothing from the people who left gets in."

The recruiting path reaches a particular group.

"The response rate is 3 percent. Hard to speak for everyone from that."

Who answered may differ from who didn't.

"Let's set the outlier rule before we open the data."

Defining the filter after seeing the results moves the conclusion.

After that episode, Pagedeck split the satisfaction survey in two. The in-app banner stayed, and churned customers got asked separately by email. The email path returned only 11 percent, but what came back from that 11 percent was different: half of them said they left because permission controls weren't good enough.

The report format changed too. Next to the score, it now carries the number of respondents, the response rate, and the plan composition of the respondents. Mia asked for those columns because that's what tells her how far the conclusion reaches.

The most common belief is that a bigger sample fixes it. Everyone has heard that larger samples are better, so the conclusion is natural. But size and direction are separate problems. Collect ten times as much through the same skewed path and the same conclusion merely looks sturdier, which makes it more dangerous — a report with 4,000 responses is harder to argue with than one with 412. What needs increasing is the number of recruiting paths, not the number of responses.

The second is assuming random sampling removes it. Randomizing who gets contacted does close the first door. Who among them replies is still their own choice. Mail 10,000 people at random, get 300 replies, and those 300 are not a random sample. Non-response is a second selection step, and usually the larger of the two. So even after random selection, response rate and respondent composition have to be read alongside the result. Skip that and the fact that the sample was drawn at random becomes the reason people over-trust the finding.