Causation

As records pile up, finding two numbers that move together has become trivial. User behavior is logged across dozens of fields, so pick any two, lay them over each other, and a pair that rises and falls in step turns up almost immediately. The trouble starts after the pair is found.

What happened at Pagedeck is typical. Accounts that invited three or more colleagues in their first week were twice as likely to still be around six months later. So the team made the invite screen impossible to skip and required three names before a new user could continue. Two months later the six-month retention rate was unchanged. Teams that invited people weren't retained because of the invites; teams that had already decided to use the tool together both invited more people and stayed longer.

The word causation exists to keep those two apart. "These two numbers move together" and "move the first one and the second follows" are claims of completely different size. The first can be read off a chart; the second usually requires an experiment. The line where expensive decisions get reversed in a meeting is almost always this one.

Calling A the cause of B requires three things at once.

1. Order in time — A has to happen before B. Whether accounts that stayed invited more people, or accounts that invited more people stayed, gets screened first by the order in the log.
2. Movement together — B has to change when A changes. This is correlation, the degree to which two values move in step. It's a necessary condition, and on its own it settles nothing.
3. Ruling out other explanations — no third factor can be driving both values. In the example above, the team's intent to adopt was that factor. A hidden value that makes a relationship look real this way is a confounding variable.

The third is always the hard one, because you can only check the hidden factors you managed to think of, and there is no way to check the ones you didn't. So instead of eliminating them one at a time, the standard move neutralizes all of them at once: split people into two groups at random.

Random assignment spreads intent to adopt, team size, industry, and everything nobody thought of evenly across both groups. Apply a change to one side only, and the change is the sole remaining explanation for whatever difference shows up afterward. That is what an A/B test does, and the side left untouched as the basis for comparison is the control group. You can never see what the same person would have done without the change, so a statistically equivalent group of other people stands in for that unseen case.

Often you can't split at all. Prices can't vary by user, and events that already happened can't be rerun. Then the fallback is staggering a rollout by region or comparing the periods before and after a change. Those designs get mixed up with everything else happening in the same window, which makes the conclusion one notch lighter. That the screen changed in March and conversion rose in March doesn't rule out the ad campaign that also ran in March. The usual repair is to find a comparable group that got none of it — another region, another segment — and check whether their numbers moved too.

Four places where this distinction turns into money.

1. Shipping decisions — confirm with a randomized comparison that a new screen really lifts conversion before turning it on for everyone. Turn it on without that and the effect can't be separated from seasonality or a campaign running the same weeks.
2. Marketing budget — switch one channel off in a single region for a few weeks and watch whether signups in that region actually drop. Signups rising while the channel ran says nothing about what the channel contributed.
3. Choosing target metrics — before making a leading indicator, a number that moves ahead of the result, into a team goal, check whether it drives the result or merely rises alongside it. Target the second kind and the metric climbs while the result stays put.
4. Incidents and anomalies — when a metric drops on a day that had both a deploy and an outage upstream, comparing against an unaffected segment narrows the cause.

"That's correlation, not causation. Let's run a test before we commit."

A proposal to confirm with a randomized comparison rather than decide from observation.

"Accounts that invite more do stay longer, but forcing invites didn't make anyone stay."

Forcing the first of two co-moving values didn't bring the second along.

"Signups were up last month, but the campaign was running then too."

Two candidate causes overlap in the same window, so their contributions can't be separated.

After the invite experiment ended, what Mia did was change the question. Instead of looking for ways to raise the invite count, she asked what the teams who invited heavily had come to do in the first place. The sales notes showed those teams arrived intending to keep meeting records together, and the invites were a consequence of that intent.

So the next experiment targeted the post-signup screen, where a team states what it plans to use the product for, rather than the invite button. Half the new accounts saw that screen and half did not. The gap in four-week retention between the two groups became the basis for the decision. The sentence Mia repeats in meetings comes from the same place: two numbers moving together tells you where to dig next, not what to change.

The most common mistake is treating a strong correlation as good enough. When two lines nearly overlap, coincidence feels implausible, so the inference comes naturally. But the strength of a relationship and the direction of it are separate questions. A tighter overlap also raises the odds that a single common cause sits behind both. Force a feature on the strength of a correlation and you spend a quarter the way the invite change did, ending where you started, with a worse signup flow than before.

The second is reading a completed experiment as proof of causation in general. A randomized comparison is the strongest evidence available, but its conclusion holds only inside the range it was run in. A two-week test on new signups tells you what happened to new signups over two weeks; long-tenured users and six-month effects are not in it. Variants that win a short test sometimes lose money over a year, which is why teams keep a holdout, a group left untouched for a long stretch, to read the slower effect.