Confidence Interval

Numbers in a report usually come from a slice rather than the whole. You can't ask all 120,000 users, so you ask 800 and write their answer down as the answer. But those 800 are a different 800 every time they are drawn, and a new draw gives a slightly different result. A report carrying one number leaves no trace of that wobble.

So meetings kept stalling in the same place. Satisfaction goes from 44 percent last quarter to 46 percent this quarter, and someone asks whether asking again would produce 46 again. Nobody can answer, so the room splits between people reading a two-point gain and people reading no change at all, and the loudest reading wins.

A confidence interval is an agreement to write the size of that wobble next to the number. Instead of 46 percent you write 42.6 to 49.4 percent. A wide range means there is still little you can say, and a narrow one means you can say it plainly.

Take the value from the sample, measure how far it moves from sample to sample, and add that much on each side.

A sample here means the slice drawn to stand in for everyone. 800 people out of 120,000 are the sample, and the confidence interval is the range attached to speaking about 120,000 from those 800 answers.

There are four steps.

1. Get the value from the sample. If 368 out of 800 say they are satisfied, that is 46 percent.
2. Measure how far that value moves between samples. For a proportion, multiply 0.46 by 0.54, divide by 800, and take the square root. That comes to about 0.018, or 1.8 percentage points — roughly how far the answer shifts each time a new 800 is drawn.
3. Pick a confidence level and multiply by its factor. At 95 percent the factor is about 1.96, and 1.8 times 1.96 is 3.4 percentage points.
4. Subtract and add that much from the center. 46 minus 3.4 is 42.6 and 46 plus 3.4 is 49.4, so the interval runs from 42.6 to 49.4 percent.

A report writes this as 46 percent (95 percent confidence interval 42.6 to 49.4).

Only two things set the width: sample size and confidence level. A larger sample narrows it, but not in proportion to the headcount. Step 2 puts the sample size under a square root, so halving the width takes four times the people. The 3.4 percentage points from 800 respondents becomes 2.4 at 1,600, and reaching 1.7 takes 3,200. The confidence level decides in advance how often you are willing to be wrong. 95 percent is the convention, and it gets fixed before the survey runs rather than moved after the result is in.

Narrower is better, but no width counts as passing on its own. The test is not the width itself — it is whether a decision can be made at that width.

Read each end of the interval separately. If both ends point to the same decision, the interval is narrow enough. Say the call on keeping or dropping a paid plan sits at 45 percent satisfaction: an interval of 42.6 to 49.4 percent says drop at the low end and keep at the high end, so it can't be used yet. With the line at 35 percent, both ends say keep and the decision is ready.

When comparing two points in time or two variants, check whether the intervals overlap. Last quarter at 40.3 to 47.7 and this quarter at 42.6 to 49.4 overlap heavily, so writing that satisfaction went up is off the table. A gap between 44 and 46 is the size that shows up from asking the same group twice.

Different metrics need different widths. Conversion rate, the share of visits that produce the action you wanted, turns 0.1 percentage points into revenue, so it has to come down near 0.2 percentage points before it is usable, while a satisfaction survey read only for direction survives at 4. That is why an A/B test — splitting traffic randomly between two variants and comparing the results — starts by working out the required sample size. Fix the difference worth detecting and the headcount for an interval that narrow follows from it.

"Variant A converts at 3.1 percent and B at 3.4, but the intervals overlap."

The ranges sit on top of each other, so B can't be called better yet — collect more or shut the test down.

"Only 120 people answered, so the interval runs 9 percentage points either way. We can't decide on this."

The number exists; it just can't be used.

"Put the interval next to that 46 percent."

A lone point tells the reader nothing about how much it wobbles.

At Pagedeck the range comes up first whenever the quarterly survey results land — the same questions put to many people to see the distribution. Mia brings in satisfaction at 46 percent, Grace compares it against 44 from last quarter, and Mia puts the two intervals side by side, points out how far they overlap, and proposes writing "level with last quarter" instead of "up."

When Theo suggests collecting more responses, Mia answers that four times as many are needed to halve the width. Getting 3,200 answers would stretch the survey past a month and the product changes in the meantime, so that quarter gets judged on something other than satisfaction.

The most common mistake is reading a 95 percent confidence interval as a 95 percent chance that the true value sits inside it. The word confidence sitting next to the number 95 invites exactly that reading. What the 95 percent attaches to is not the range in your hand but the procedure that produced it: draw 100 samples the same way, build 100 intervals, and 95 of them contain the true value. Whether this particular one is among the 95 is unknowable. Left in place, the reading leads to ranking positions inside the interval and treating the middle as the likelier answer.

Reading a higher confidence level as a more accurate result is the other common one. 99 sounds better than 95, so it sounds like better analysis. All that changes is the factor in step 3, from 1.96 to 2.58, which widens 3.4 percentage points to 4.5. The risk of being wrong drops, and so does the amount that can be said. Left alone, "we ran it at 99 percent" circulates as the more trustworthy report while the fact that nothing can be decided across that wide a range goes unmentioned.