Simpson's Paradox

Numbers don't have to lie for the way they're grouped to change the answer. That two opposite statements can both be true of one dataset is hard to believe before you've met it, and easy to walk past afterwards.

Pagedeck's signup screen experiment was the introduction. Ravi ran the existing screen against a new one for two weeks. Overall, the existing screen converted at 8.0 percent and the new one at 6.2 percent. The meeting was heading toward retiring the new screen when Mia asked to split by device. Split, desktop showed 10.0 percent for the existing screen against 12.0 for the new one, and mobile showed 4.0 against 5.0. The new screen won in both groups and lost in the total.

The arithmetic was not wrong. A mobile campaign had been running during the test, so a far larger share of the people who saw the new screen were on mobile, where conversion is less than half of desktop. What flipped the conclusion wasn't the screen, it was who saw it — a fact that lived in the campaign calendar rather than in the experiment report.

It appears when three conditions coincide: the population is mixed, the baseline differs by group, and the group composition differs between the two things being compared.

Laid out with Pagedeck's numbers:

1. On desktop, the existing screen converted 200 of 2,000 people, which is 10.0 percent, and the new screen converted 60 of 500, which is 12.0 percent.
2. On mobile, the existing screen converted 40 of 1,000, which is 4.0 percent, and the new screen converted 125 of 2,500, which is 5.0 percent.
3. Combined, the existing screen is 240 of 3,000, or 8.0 percent, and the new screen is 185 of 3,000, or 6.2 percent.

The total is a weighted average of the group values, where the weights are how many people landed in each group. Two-thirds of the existing screen's audience was desktop, where conversion is high; five-sixths of the new screen's audience was mobile, where it's low. With weights that different, the group ordering can hold while the total ordering inverts.

Which figure to trust depends on what you split by.

1. If the splitting variable was already determined before the comparison and it also affects the outcome, the split figures are the ones to trust. Device is fixed before anyone sees a screen and it affects conversion. A variable like that is a confounding variable.
2. If the splitting variable is part of what happened after the comparison, the total is the one to trust. Compare document creation among only the people who signed up, and you've filtered by a characteristic the new screen itself influenced, so the split tells you nothing about the screen. The test is simple: was this value already fixed before the comparison started?
3. When the call isn't obvious, go back to the design. Assign people to the two screens at random and the compositions match by construction, so the effect never arises. That's why a check on device composition belongs right after assignment — if it's skewed, fix the assignment before reading any result, and treat the numbers collected so far as unusable.

Anywhere a comparison happens. Four places most often.

1. Reading experiment results. When the two groups of an A/B test weren't split at random, or a particular source of traffic piles into one of them after the split, the conclusion can invert — and the report will look perfectly ordinary.
2. Comparing acquisition channels. Two ad channels compared on conversion can't be settled from totals when one of them carries mostly first-time visitors. Different channels bringing different kinds of people is the baseline condition of that comparison.
3. Comparing periods. Grouping people by when they arrived — cohort analysis — is partly a defence against this. Overall retention falls whenever new signups rise, with no change in the product.
4. Comparing teams or regions. Two teams compared on success rate invert when the difficulty mix of their work differs: the team handling the hard cases can be better at every category and lower overall, which is how a performance review reaches the wrong conclusion from correct data.

The method for catching it is fixed: when the total contradicts expectations, split by the main variables — and decide that list in advance. Pagedeck settled on device, acquisition channel and new-versus-returning, and attached all three to every experiment report. Without a predetermined list, splitting only happens when someone dislikes the conclusion, and that isn't analysis, it's shopping for an answer.

"Overall the new screen is lower and by device it's higher in both."

Reporting that the total and the splits point opposite ways.

"The mobile share differs between groups, so we can't compare them as they are."

Flagging mismatched composition.

"Let's fix the split variables before the experiment starts."

Preventing the choice from being made after seeing results.

When Mia and Ravi re-ran that experiment, they assigned within device: half of desktop to each screen, half of mobile to each screen. Two weeks later the overall figures agreed with the groups — 9.3 percent for the new screen against 7.8 for the existing one. The first experiment's conclusion wasn't wrong so much as its assignment was, and what surfaced the difference was one table with a split in it.

The most common reaction is that splitting always gets closer to the truth. After meeting this once, people stop trusting totals. But choose the wrong variable to split by and the split figures are the distorted ones. Split by something produced after the comparison and you filter on a characteristic the treatment caused; keep splitting and each cell gets small enough to support any conclusion you like. A split is a hypothesis rather than a tool, and which variables you'll split by belongs on paper before the comparison begins, alongside what result would change the decision.

The other belief is that this is a textbook curiosity. It has a name and the examples are tidy, so it feels rare. In practice, comparing two groups with different compositions happens daily: a week with a campaign against a week without one, a month heavy with new customers against one that isn't, two experiment arms with different device mixes. Remembered as a special event it slips past every day. Turned into a habit — split whenever the total contradicts the direction — it gets caught the same afternoon, usually in the first table someone builds.