Confounding Variable

Once records pile up, values that move together start standing out. Accounts that invited more than three colleagues are still around six months later 62 percent of the time, against 31 percent for accounts that invited fewer. After a gap that wide, the next sentence writes itself: push people to invite more and more of them will stay.

Pagedeck spent a quarter on that sentence. An invite banner went on the first screen, reminders went out, invite counts climbed — and six months later retention had not moved. Splitting accounts by team size afterward changed the picture. Larger teams were the ones doing the inviting and the ones staying, and team size had been sitting between the invite count and the retention number the whole time.

When the same thing happened with channel performance and with satisfaction by plan, the value doing the interfering needed a name. Confounding variable is the single phrase for saying that something outside the table is moving two of its columns, and that the relationship can't be read as cause until that something is found.

When a third value feeds both the cause column and the effect column, the two columns move together even though nothing connects them.

Confounding requires three conditions at once.

1. The value moves the supposed cause. A team with more people has more colleagues to invite, so the invite count rises.
2. The value moves the effect separately. Larger teams adopt a tool as an organization, so the account survives one person losing interest.
3. The value is not a step between cause and effect. If inviting people is what made the team larger, this is the path the cause travels rather than a confound, and the banner really would raise retention. So it matters whether the third value existed before the cause or appeared after it. Team size was set by hiring long before anyone opened the invite screen, which puts it on the confound side.

The quickest check is to split the data by the suspect value and run the same comparison inside each slice. Splitting Pagedeck's 62 against 31 by team size gives 64 against 58 among teams of ten or more, and 29 against 26 among teams under five. When the gap nearly disappears after splitting, team size made it, not invites. The case where splitting flips the conclusion outright has its own name, Simpson's paradox.

It appears wherever someone wants to claim a cause from data where groups were not split at random. Five situations account for most of it.

1. Feature use tied to retention. The observation that people who used a feature stayed longer also contains the fact that they were engaged enough to go find it.
2. Acquisition channel comparisons. Search traffic converts better than banner traffic, but people who searched had already felt the need that sent them to the search box.
3. Customer segment comparisons. Enterprise accounts reporting higher satisfaction come bundled with contract size, dedicated support, and a different reason for buying.
4. Before-and-after release comparisons. If the metric rose the week a feature shipped, the campaign that ran that week, the season, and a pricing change are all candidates.
5. The measured effect of customer success outreach. Accounts with an assigned manager renew more often partly because large accounts are the ones that get a manager.

Two defenses exist and they are not equally strong. The reliable one is randomization, the procedure for splitting groups without bias. Assign by coin flip and team size, acquisition channel, and values nobody has thought to name all spread evenly across both sides. That is what gives an A/B test, which splits traffic randomly between two variants and compares results, the right to talk about causes at all.

When random assignment isn't available — analyzing records that already exist, for instance — splitting stands in for it. But splitting only defends against values you can name. A value with no column in the table can't be split on, so any conclusion drawn from observed data carries unknown confounds. A gap that survives splitting isn't proof there is no confound; it means the values you know about don't explain it, which is a weaker claim than it sounds.

"Could this be a confound? Let's split it by team size and look again."

The visible gap may belong to another value, so measure it again inside slices.

"Groups weren't assigned randomly, so we can't call this difference a feature effect."

Not doubting the number, just limiting what it can be named.

"The release and the campaign went out the same week. There's no way to tell which one did it."

Two changes overlap and nothing separates their contributions.

At Pagedeck the phrase comes up most often in quarterly review. Mia reports that the invite banner doubled invites while retention stayed flat, and Theo suggests splitting last quarter's 62 against 31 by team size. The split table shows gaps of 4 and 3 percentage points, and the original hypothesis falls apart on screen.

The conclusion usually takes the same shape. Whether invites drive retention can't be answered from these records, so if the answer matters, next quarter gets a randomized test. Until then the decision about keeping the banner stays open, and the 62 percent figure stops being quoted in decks as evidence that invites work.

The most common belief is that more data fixes it. Small samples producing strange results is a familiar experience, so this looks like a problem of size. But size reduces chance, not bias. Data with team size mixed into it is equally mixed at 5,000 accounts and at 50,000, and the narrower range that comes with more data only makes a wrong conclusion feel more certain. Left uncorrected, this belief costs a quarter of collecting and lands in the same place, with a tighter estimate of the same distorted number.

The other common belief is that controlling for every variable that comes to mind is enough. Analysis tools accept many variables, so adding more feels like cleaning the estimate. But only values that are named and recorded can go in, and things like what a user's job actually involves, or whether one person pushed the tool through internally, stay outside the table. So "probably" is the honest word for a result from observed data, and calling it causation takes at least one randomized test.