Diagnostic Analysis

A weekly metrics review opens with a slide: paid conversion after signup fell from 12 percent to 8 percent. That much is the work of descriptive analysis, which summarizes what happened in numbers. The meeting stalls on the next question — why.

The gap usually gets filled with guesses. Marketing thinks the traffic mix changed, design suspects the signup screen that shipped last month, engineering wonders whether the payment integration is throwing errors. All three sound plausible and none of them carries evidence. The next month then gets spent on whoever argued loudest.

Diagnostic analysis fills that gap with data. Instead of staring at one aggregate number and brainstorming causes, it cuts the records behind that number by condition and looks for the slice where the change actually landed. If 8 percent turns out to be the midpoint between 19 percent on team plans and 3 percent on individual plans, and only individual plans dropped, there is one suspect instead of three.

Write the change down as a sentence, choose the axes to cut by, compare the slices, and verify what survives — four steps.

1. State what changed and by how much in one sentence. Not "conversion got worse" but "paid conversion within 7 days of signup fell from 12 percent in August to 8 percent in September." The period and the population have to be pinned down or the next step has nothing to compare against.
2. Choose the axes. An axis is a value already sitting in your records — plan, traffic source, signup cohort, device, country. You cannot cut by a value nobody recorded, so how far event tracking reaches sets the ceiling here. Event tracking is the practice of logging which buttons people pressed and which screens they saw.
3. Cut along each axis and look for the slice that dropped. If the whole fell 4 percentage points and no slice fell anything like that much, that axis is not the cause. If one slice sank from 15 percent to 5 percent, that slice moves on.
4. Cut the surviving slice by stage. Funnel analysis lays the stages from signup to payment in order and counts how many people remain at each one, so knowing which stage leaked narrows the candidates again. If individual-plan users drop out only at the card entry screen, there is exactly one screen to look at.

All of that narrows candidates, and whether the surviving candidate is the cause is a separate question. The cleanest way to answer it is an A/B test that reverts just that piece — an experiment that shows two versions to randomly split groups and compares the results. When the change cannot be reverted, put a comparable group that didn't go through it side by side instead.

It fits when the change has already happened and the cause needs narrowing: a metric bending visibly, one segment behaving unlike the rest, an experiment landing far from what was expected. What those three share is that the difference already sits in the data, waiting to be sliced out.

The misfits are just as clear. First, a cause that was never recorded never turns up. A competitor cutting prices in half last month appears nowhere in your logs, so every cut returns the same answer — everything fell evenly. That is the moment to ask customers directly rather than run another query.

Second, numbers get shaky as slices shrink. Cut 4,000 people by plan, then traffic source, then device, and a cell holds 30 people, where two conversions either way swing the cell by 7 percentage points. That is noise, not a cause. Decide the minimum cell size before you start cutting.

Third, moving together says nothing about why. Correlation is the degree to which two values move together and causation is one of them actually producing the other, and cutting data only ever shows the former. A confounding variable can sit between them — a hidden value that affects both and makes the relationship look real. If individual-plan conversion fell because of the ad traffic that happened to land there rather than the plan itself, fixing the plan brings nothing back.

"Let me run a diagnostic on the conversion drop before we talk. I want to know which stage it is."

Settle the cause by cutting the data, not by guessing.

"I layered three axes and now there are twenty people per cell."

The slices got so fine that the differences are no longer trustworthy.

"We have the diagnostic, but whether that's the cause still needs an experiment."

Drawing a line between a narrowed candidate and a confirmed cause.

At Pagedeck this is the first thing scheduled in a week where a metric bends. Mia pins the drop down in one sentence and shares it, Ravi cuts by traffic source, and Theo walks the payment logs stage by stage. Last quarter it took exactly that to show the drop sat entirely with users from one ad channel. The debate about the signup screen ended there, and what got fixed was the channel's targeting, not the screen. Mia asks for the size of each cell first, because a difference built on twenty people is one the next week can reverse.

The most common mistake is believing that diagnostic analysis delivers the cause. The name says diagnostic and the output points at one place — "individual plans dropped" — so it reads that way. But cutting data shows where a change is concentrated, not the force that produced it. Confirming a cause takes a separate step: revert it, or set a comparison group beside it. Skip that distinction and a team spends a month fixing the place the change was concentrated, watches the number stay put, and stops trusting data at all.

Assuming a single cause is just as common. One slice usually looks worse than the others, so that is where people stop. More often the 4-point drop is a 2-point change in ad traffic quality, a 1-point payment error, and a 1-point seasonal effect stacked together. Fix one and the number comes back halfway, with the other half unexplained and carried into the next month's review.