A/B Testing Tool

For a long time, decisions about what a screen should say were settled in a meeting room. Five people would argue over whether the pricing page button should read "Start for free" or "Try 14 days free," and the version that shipped was usually the one proposed by whoever was most senior or most insistent. Nobody could confirm the choice was wrong, so the same argument came back a month later.

Teams did try to settle it with numbers. Run the old copy this week, swap in the new copy next week, compare conversion rate — the share of people who arrive on a screen and go on to do the thing the screen asks for. The trouble is that the two weeks are not the same conditions. If the ad budget went up in week two, or a holiday fell in the middle, that effect is tangled up with the copy change, and nobody can say what the extra 0.4 percentage points came from.

Splitting people in half during the same window removes the tangle. What it leaves behind is the work of splitting and counting, and that work is harder than it looks. If someone sees a different screen on every refresh, the results turn to mush. If one group ends up with 2,300 people and the other with 1,900, comparing raw rates tilts the answer. The last step is the worst. Whether 3.1 percent versus 3.4 percent counts as a real difference depends on how many people came through, and teams doing that math by hand usually get it wrong.

The tool takes over three jobs.

1. Splitting — every arriving visitor gets assigned as if by a coin flip. Split by a rule instead, like morning visitors into A and afternoon visitors into B, and the two groups differ as populations, so any gap that shows up could be the screen or could be the people.
2. Holding people in place — once someone lands in group A, they keep seeing A on refresh and three days later. The tool recognizes them by a marker left in the browser, which means a private window or a second device turns the same person into a new one.
3. Deciding — it calculates statistical significance for you. Statistical significance is the line between a gap that chance alone could produce and one that it probably couldn't. The tool takes the headcount and conversions for both groups, returns that verdict, and when the test is still too early, tells you how far you are from the required sample size.

Record-keeping comes along with it. Which variant ran, from when to when, in front of how many people, with what result — all of it accumulates without anyone writing it down. Six months later, when the same idea comes back around, that log is the evidence that it was already tried. Keeping one person out of two overlapping experiments is also the tool's job.

You can run experiments without buying a tool. Three approaches are common.

1. Split with a feature flag — a feature flag is a switch that turns a feature on or off in code that's already deployed. If your team already uses them, you can halve user IDs and turn the new screen on for one half. That covers the split but not the verdict, so results have to be counted by hand in a web analytics tool or an analytics store.
2. Log the assignment and let an analyst do the math — record which group each person landed in, and an analyst can later compute per-group rates and significance together. The quality of the verdict is highest here, but each test costs several analyst days, which caps most teams at one or two a month.
3. Change it and compare before and after — the most common approach and the weakest. Comparing the week after against the week before lets seasonality and campaigns back in. That said, if a page gets 200 visits a day, splitting it never accumulates enough people for a verdict, and before-and-after is what's left.

The choice usually comes down to frequency. A team that reworks a screen once a quarter won't get its money back from a tool; a team shipping tests weekly finds waiting on an analyst more expensive.

"Let's put both versions of this line in the A/B testing tool and look next week."

Stop deciding in the meeting and wait until there are numbers.

"The tool still says it isn't significant. We need about three thousand more people."

The gap on screen is small enough that chance could produce it, so no winner yet.

"We can't split this one in the tool. It's behind login, so it has to happen server side."

Swapping visible elements with a script on the page doesn't reach this part of the product.

At Pagedeck, Ravi opens this tool more than anyone else. When someone suggests changing the opening line of the landing page, he loads both versions, splits traffic evenly, and then leaves the page alone until the verdict arrives. He leaves it alone on day two even when one side is clearly ahead, because early numbers swing hard, and stopping there means crowning whichever version got lucky.

Once the verdict lands, it moves to Mia. She decides whether the winning line belongs on screens inside the product too, or whether the result only describes people who arrived from ads. The tool answers which side scored higher and stops there.

The most common mistake is believing that installing the tool starts the experimenting. The dashboard shows a list of tests and a win rate, so it looks that way. What the tool actually handles is splitting and deciding; what to change and why is still a human question. Keep changing button colors without a hypothesis and the dashboard fills with forty tests, two or three of which reached significance, and even those come with no explanation of why they won, so nothing builds on them.

Checking every day until a winner appears is the other habit worth breaking. The numbers update live, so watching them feels natural. But the verdict is built on the assumption that you look once, after the planned headcount has arrived. Watching daily and stopping the moment it flashes significant locks in whichever day happened to spike. Winners picked that way tend to fade back to the old rate after rollout, and once that happens twice, the team stops believing the tool at all.