Sample Ratio Mismatch

There is exactly one reason to believe an experiment's result: the two groups being compared are alike on average except for the thing you changed. In an A/B test, where two versions run side by side, that likeness comes from randomization. Split people by something close to a coin flip and new and existing users, mobile and desktop, end up mixed into both groups in similar proportions.

The problem is that assignment breaks quietly. If the new screen fails to render at all in one browser, the people using that browser were assigned to the new screen and then vanished without logging anything. What's left in the results table for the new screen is only people who don't use that browser, and the two groups are no longer alike.

Experiments like that used to get called anyway. The results screen showed two conversion rates and a winner, so nobody checked how many people had landed in each group. The conclusion was that the new screen won, when what actually happened was that users with a broken experience dropped out and their absence got read as an effect.

Sample ratio mismatch (SRM) is the check that catches this before a call gets made. Confirm the counts match the intended split first, and if they don't, don't read the result at all.

The check is one sentence: compare the intended ratio against the counts you actually got, and ask whether the difference is explainable by chance.

The hard part is where chance ends. A random split never lands exactly even, so there is always a small gap, which means the gap has to be judged together with the sample size. In an experiment with 1,000 people, 520 against 480 is ordinary noise. In one with 200,000 people, 101,000 against 99,000 is the same half a percentage point and is a number chance almost never produces.

The call gets made with a p-value — the probability of seeing a gap this large from chance alone. Feed in the expected ratio and the observed counts, and a very small value means assignment is broken. The threshold is far stricter than the one used to judge the experiment's own result, often below 0.001, because every alarm costs you an experiment and nobody wants to discard healthy ones.

Five causes account for most mismatches.

1. The assignment mechanism itself. The same user falls into different groups on different visits, or a cache serves one version more often than the other.
2. Missing logs on one side. The browser case above lives here. Assignment worked, but event tracking — the step that records user behavior — captured fewer of them.
3. Filtering applied after assignment. Assignment covered everyone, but the analysis keeps only logged-in users, so any difference in login rates between groups shows up as a count gap.
4. Traffic that isn't people. Crawlers or automated monitoring tools land disproportionately on one side, which is easy to miss because they behave like very consistent users.
5. Rules changed mid-flight. Adjust the split from 50/50 to 30/70 while the experiment runs and the totals match neither setting.

Which side came up short helps narrow the cause. Between the control group and the treatment group — the one left alone and the one seeing the change — a shortfall only on the treatment side points at something failing in the new experience, while both sides looking odd points at the assignment layer.

The check sits in four places.

1. Automatic warnings in the experiment tool. Many platforms flag the results screen and lock the winner call. It's designed to catch you whether or not you thought to look.
2. First item in the readout meeting. Read the counts before the conversion rates. Reverse the order and the result you've already seen becomes hard to throw away, which is when someone suggests interpreting a broken split anyway.
3. Release configuration checks. A misconfigured feature flag — the switch that turns a shipped capability on and off — skews the ratio, so the flag setup is the first thing to inspect when an alarm fires.
4. Data pipeline checks. The signal is useful beyond experiments: one side logging less means the tracking code on that screen is broken, which remains to be fixed after the experiment is abandoned.

When an experiment gets discarded, record that in the experiment log. There is no result to write down, but the same cause resurfaces in the next experiment, and the note is what stops a second team from losing two weeks the same way.

"The split came out 51 to 49. Let's check assignment before we look at the result."

Counts are off, so hold off on interpreting conversion rates.

"This one has SRM, so we're discarding it, fixing the cause, and rerunning."

Nothing here is salvageable; the experiment starts over.

"With only 1,200 people, 52 to 48 is inside the normal range."

The same ratio gap is explainable by chance at a small sample size.

Pagedeck ran a document template recommendation experiment last quarter. When Mia opened the results two weeks later, the control group held 128,000 people and the treatment group 120,000. The 6 percent gap stood out, and conversion looked higher in the treatment group.

Theo spent a day on the cause. The request that loaded the recommendation list failed often on slow connections, and users whose request failed closed the screen without logging anything. Effectively only slow-connection users dropped out of the treatment group, so the apparent win could simply have been the disappearance of people who convert less to begin with. The experiment was discarded, and the rerun after the fix showed no difference.

The most common mistake is assuming a small gap can be corrected arithmetically. The ratio is known, so weighting the smaller side looks reasonable. But correcting requires knowing who dropped out, and if you know that you can fix the cause instead. The people missing tend to share a property — slow connections, old app versions, conditions that already convert poorly. Comparing averages with that group erased mixes the effect of your change with the effect of their absence, and a win declared on that basis fails to move the metric once it ships to everyone.

Reading a clean ratio as a healthy experiment is the other error. This check only looks at whether assignment and logging broke. Even with the counts exactly even, the peeking problem — calling a result at the moment one side happens to be ahead — is entirely untouched. Checking the ratio is the first gate on whether the result may be read, and nothing more than that.