Incrementality

Ad performance was treated as a counting problem for a long time. Someone clicked an ad, later paid, and the payment got recorded as something the ad produced. Deciding how to count was the job of attribution. The trouble is that the record is not causation. Someone who has already decided to buy searches for the company name, clicks the ad at the top of the results, and signs up. On paper that's an ad-driven conversion. In reality, with the ad off they would have clicked the organic result one line below and paid anyway. The money left the account and the report counted it as a win.

The same scene kept repeating in review meetings. Add up what each channel claimed and the total ran higher than the number of new customers that month, because one person saw a banner and clicked a search ad and every channel wrote the whole conversion into its own column. Then the founder would ask the obvious question: if we turn this ad off tomorrow, how much revenue do we lose? The records hold no answer, because nobody ever turned it off.

Incrementality is the practice of deliberately producing that missing number. Split the audience into a group that keeps seeing the ads and a group where the ads are switched off, then credit the advertising with nothing more than the difference between the two. The question shifts from who clicked to whether the presence of the ad changed the outcome at all.

Build two comparable groups, run the ads to only one of them, and treat the gap in conversions as the net lift the spending created. The structure is the same as an A/B test — split people at random, apply something to one side, compare results — except that what differs between the sides is ad exposure rather than a screen.

There are four steps.

1. Split the audience at random. The side that keeps receiving ads is the exposed group; the side with ads switched off is the control group, and a control group held empty over a long stretch like this is called a holdout. Splitting by region or age instead makes the two groups differ in disposition, which breaks the comparison before it starts.
2. Count conversions in each group over a fixed window. What counts as a conversion comes from your conversion tracking setup. Change that definition mid-experiment and the two groups are no longer being measured with the same ruler.
3. Take the difference. If 500,000 exposed people produce 12,000 signups and 500,000 control people produce 9,000, the net lift is 3,000. The other 9,000 would have arrived without ever seeing an ad.
4. Turn it into a ratio. Dividing the 3,000 of net lift by the 12,000 recorded conversions gives 25 percent, meaning only a quarter of the credited performance was actually produced by the ads.

Recalculating cost with that number changes the picture. If the campaign spent $27,000 over the window, the recorded customer acquisition cost (CAC) is $2.25. Divided by net lift instead, it's $9.00, and whether to raise the budget gets decided on that second figure.

A single number tells you nothing about high or low on its own. The same 25 percent means opposite things depending on whether it came from ads shown to people who already know the company or ads shown to people meeting it for the first time. The metric earns its keep in comparisons between channels.

Channels that reach people who already know you produce lower numbers. Search ads placed against your own company name routinely land below 10 percent, because switching them off just routes the same person to the organic result underneath. Channels reaching people who have never heard of you can clear 40 percent. Two channels can both report a return on ad spend (ROAS) of 500 percent while one is quietly leaking money and the other deserves more budget.

Two things decide whether a result is trustworthy. The first is whether the gap is wide enough to be hard to explain by chance. A difference between 12,000 and 11,900 could flip on a rerun, so nobody claims a lift before checking statistical significance. The second is duration. If the product takes two weeks to decide on, the experiment runs at least that long. Cut it off after a week and the slow arrivals in the control group haven't landed, which inflates the lift.

Measurement itself costs money. The group with ads switched off gives up real revenue for the duration, so teams usually start with the channels taking half the budget rather than the small ones.

"This channel's ROAS looks great but there's no lift. I think it's because it's brand search."

The recorded revenue would have arrived even with the ads off.

"Before we raise the budget, let's run a two-week holdout."

Hold part of the audience back, see the difference, then decide.

"Recalculated on last quarter's lift, each customer costs us nine dollars."

Divided by net lift instead of recorded conversions, the real cost is four times higher.

At Pagedeck the term first came up over brand search ads. In Ravi's monthly report that channel always looked like the strongest performer, since people searching the company name clicked through and signed up at a high rate. Grace asked whether those people would stop arriving if the ads stopped.

Ravi switched the ads off in half the regions for two weeks. Signups fell 4 percent, while the channel had been claiming a third of all signups in the reports. The team cut the budget to a third of what it had been and moved the rest to channels that reach people who don't know Pagedeck. Two months later total signups were up and total ad spending was unchanged.

The most common reaction is that the method undercounts conversions. If net lift is a quarter of the recorded number, it looks like three quarters are going unmeasured. But those three quarters aren't missed results — they're results that would have happened anyway. Nothing failed to get counted; it simply was never the ads' to claim. Teams that miss this distinction conclude their measurement is broken whenever lift comes back low, and end up replacing tracking tools while the leaking budget keeps leaking.

The second mistake is treating the number as a one-time reading. Experiments are expensive, so there's real pressure to keep using a figure once you have it. But lift moves with market conditions and with how well known you are. When nobody has heard of you, brand search ads show high lift; once the name is familiar, the same ads show much less. Plan this year's budget on last year's figure and you keep funding an ad that stopped earning its place months ago. Large channels usually get remeasured every quarter.