Experiment Toggle

When two options got decided in a meeting, the louder side won. Six months later nobody knew whether the call had been right, because neither side ever saw how the other one would have gone.

Shipping them one after the other doesn't fix it. Turn on option A this week and option B next week, and weekends, ad spend, and seasonality all land in between. Even if B's number comes back higher, there's no way to separate the screen from the people who happened to arrive that week.

An experiment toggle uses a switch that turns a feature on and off while the code stays deployed. Instead of splitting the time, it splits the people.

It solves the lack of any way to run two options under the same conditions at once. Split arriving users at random, show half the version with the switch on and half the version with it off, and the same weekdays, the same campaigns, and the same season spread evenly across both. The only difference left is the one screen.

Three things have to hold for that to work.

1. Assignment is fixed per user. If the version changes on every refresh, nobody belongs to a group and the results can't be used.
2. The split is set up front and not changed mid-run. Start at 50/50, move to 80/20 on day three, and the data from before and after can't be pooled.
3. The stopping condition is written down first — two weeks, or 5,000 users on one side. Watch the numbers and stop when you like, and you end up picking the moment that looks like a win.

When it ends, the winning version stays and the branch and the switch come out of the code. Left in, the next experiment stacks on top of this one and the two switches obscure each other's results.

Designing the comparison and judging it belong to the A/B test. The experiment toggle is the mechanism that runs that design in production code.

"Let's put this screen behind an experiment toggle and decide in two weeks."

Rather than choosing now, the team will choose once there are numbers.

"Assignment is flipping on every refresh. We can't use this data."

The groups aren't fixed, so there's no comparison to make.

"The experiment is over and the flag is still in the code."

The cleanup step got skipped, which makes the next piece of work harder.

At Pagedeck, Theo put two versions of the document sharing screen behind an experiment toggle. For two weeks half the users got the new one, and the share-completion rate came back 8 percent higher on it.

Mia rolled the new screen out to everyone and put removing the switch on Theo's list for the following week at the same time.