Minimum Detectable Effect

Run a test for two weeks and the result is often "no difference." The trouble is that the sentence covers two very different situations. Maybe the change genuinely did nothing. Maybe it did something and the test was never large enough to see it. Both produce the same-looking dashboard.

Meetings kept landing in the same place. Someone would run an A/B test, which splits users randomly between two versions and compares the results, report "5.0 percent against 5.2 percent, looks like no effect," and then someone would ask what size of difference the test could actually have caught. Nobody had that answer. Without it there was no way to tell whether 0.2 percentage points was a failure or a meaningless number, and the feature got shelved on a shrug.

Minimum detectable effect (MDE) moves that question from after the test to before it. Work the number out in advance and a small result can be written down accurately: not "this had no effect," but "this test could not tell."

Four values go into the calculation: the current baseline rate, the number of people per arm, the threshold for calling a result unlikely to be chance — statistical significance — and the chance of catching a real difference when one exists, which is statistical power. Fix any three and the fourth follows. In practice teams run it backwards: decide the difference worth catching, then solve for the people.

There are five steps.

1. Check the baseline. The usual subject is conversion rate, the share of visitors who take the action you want. Call it 5 percent.
2. Decide what size of difference would actually make you ship. A relative 10 percent — 5.0 going to 5.5 — is a typical answer, and that number is the minimum detectable effect.
3. Set the two judgment thresholds by convention: significance at 5 percent, power at 80 percent.
4. Solve for the sample size. Under those conditions you need roughly 30,000 people per arm.
5. Convert people into time. With 2,000 users entering the test each day, split evenly, each arm gets 1,000 a day, which makes this a 30-day test.

People and difference are not proportional — they're squared against each other. Halve the difference you want to catch and the people required go up four times, not two. Loosen the target in that example to a relative 20 percent and 7,600 per arm is enough, which is eight days. Tighten it to 5 percent and you need 120,000 per arm and four months. The same test can be an eight-day test or a four-month test, and step 2 is where that gets decided.

A good minimum detectable effect is not a small one. It's the size of difference that would actually change a decision. Design a test that resolves 0.1 percentage points when nobody would rearrange a screen over 0.1 percentage points, and you've spent four months buying precision you won't use. So the question that comes before the number is: if it improves this much, do we really ship it?

The second input is what the change cost. A one-line copy tweak is worth keeping even for a small win, but a payment flow rebuilt from scratch that lands a relative 2 percent gain doesn't cover its own maintenance. The effort already spent sets a floor under the number that's worth measuring.

The third is duration. If the arithmetic produces a three-month test, the number is usually unusable. Novelty effect — the short-lived bump that comes from a change simply being new — settles within about two weeks, but drag a test out longer and seasons, ad budget, and every other product change leak into the window. Past four weeks, loosen the target.

Products with little traffic can only afford large values. At 200 visitors a day, catching a relative 10 percent takes over a year. That's a case for watching people attempt the real task and finding where they stall, rather than forcing a test through. You give up a number and get a reason instead, and a reason holds up on five people.

"This test has a minimum detectable effect of 15 percent relative. Anything smaller exists but we won't see it."

A warning issued in advance that an ambiguous result will be a limit of the test.

"Dropping the MDE to 5 percent means turning a two-week test into a four-month one."

Translating a request for precision into the calendar cost it actually carries.

"Write the result up as smaller than the MDE, not as no difference."

Asking for a conclusion recorded precisely enough that the next person can dig here again.

At Pagedeck this came up first when a test was planned to cut the signup form from five fields to three. Mia put in a 5 percent conversion rate and 2,000 visitors a day, and catching a relative 10 percent came out at 30 days.

Theo pointed out that over 30 days another piece of work would land on the signup screen anyway. Mia raised the target to a relative 20 percent, which made it an eight-day test, and wrote into the plan that anything under 20 percent would be invisible to it. The result came in at a relative 24 percent.

The most common mistake is reading the value as an expected effect. A plan that says "relative 10 percent" looks like the team predicting a 10 percent improvement. But it's not a forecast, it's the resolution of the instrument. A scale that reads in 100-gram steps is not predicting that you'll gain 100 grams. Confuse the two and a result of relative 3 percent gets graded as "only a third of the goal," when the test was never able to resolve 3 percent in the first place.

Calling a below-threshold result "no effect" is the same mistake from the other side. The number sits right there, so it's tempting to judge by it, but a difference smaller than the threshold can't be separated from chance. Declaring no effect throws away changes that were working and leaves a record that discourages the next person from trying the same thing. If the peeking problem — stopping mid-test at the moment one side pulls ahead — is reading chance as an effect, this is the reverse. The accurate sentence is "this test could not resolve it."