Multivariate Test

With three elements worth changing, the default sequence is three separate two-way experiments. At two weeks each that's six weeks, and the screen changes three times along the way. Duration isn't the only problem. Measuring elements one at a time makes it impossible to see cases where the elements affect each other.

That played out on Pagedeck's signup screen. Ravi tested the headline, the button text and the presence of an explanatory image separately, and all three came back slightly positive. The month after all three were turned on together, conversion went down instead. The headline and the image ended up saying the same thing twice, which made the screen longer. Three positive results had combined into one negative one, and nothing in the three reports predicted it.

A multivariate test compares those combinations in one run. Instead of measuring elements separately it measures what they produce together, so a case where individually good changes get worse in combination surfaces inside the experiment. That is the one thing sequential tests cannot report.

Start by fixing the elements and their variants and computing the number of combinations. Six steps.

1. Decide elements and variants. Hold the elements to two or three and the variants to two each. The combination count is the product of the variants, so three elements at two variants each gives eight combinations. Writing that number down first is what keeps the design honest.
2. Put the split of the sample into the arithmetic. The minimum number of observations needed to conclude anything is the sample size; if a two-way comparison needed 20,000 people, eight combinations need 80,000. Skip that calculation and three weeks later you receive a table with no conclusion in it. The traffic the screen gets, not the number of ideas, decides how many combinations are affordable.
3. Divide visitors across the combinations without bias. That procedure is randomization, and one slot stays as the unchanged screen, the control group. Without it you can't say whether the best of eight combinations beats what you have today. The control costs one slot and answers the only question that matters.
4. Fix the duration in advance and don't look at the ranking mid-run. With eight combinations, one of them is ahead by chance on any given day, and stopping on that adopts the chance as a finding. The more combinations, the more often that temptation appears.
5. Read the result three ways: combination against combination, the average effect per element, and the influence elements had on each other. The third is the reason for running this design, so leaving it out wastes the eight-way split.
6. Turn the winning combination on for everyone, and keep the interaction you read as the hypothesis for the next experiment. Interactions are the part worth writing down for whoever runs the screen next.

Use it on high-traffic screens. Signup, checkout, the first screen — places thousands of people pass daily — are where each combination accumulates a sample. It suits cases where elements look likely to affect each other: a headline and an image filling the same region, or button text and button color read as one unit. Elements in different parts of a long page rarely interact enough to be worth the cost.

Three places it doesn't fit.

1. Low-traffic screens. Splitting the sample by the combination count stretches the time to a conclusion sharply. Compute the minimum detectable effect (MDE), the smallest difference this experiment could catch, and the required duration often comes out at six months.
2. Changes that replace a whole screen. Two screens with different structures can't be decomposed into element combinations. An A/B test, splitting traffic between two versions, is the fit there.
3. More than about five elements. Even at two variants each, that's thirty-two combinations, and at that point building the combinations blocks before collecting the sample does. Cutting elements is cheaper than cutting the conclusion.

The limits are clear. Every combination has to be built for real, so implementation cost scales with the count. Interpretation is hard too: which element helped and which combination is best can come out as different answers, so without deciding in advance what gets turned on and on what basis, the table becomes something to argue over. It also needs a rule against shipping other changes to that screen during the run. Three weeks is long enough that someone will want to.

"Three elements at two variants each is eight combinations. That's four times the sample."

The combination count sets the duration and the sample.

"This change can't be decomposed into combinations. Let's run a two-way test."

The nature of the change is what picks the method.

"The headline helps on its own, and the effect disappears in combinations with the image on."

Reading an interaction between elements.

Ravi ran the signup screen as eight combinations for three weeks, varying the headline, the button text and the explanatory image. The new headline raised conversion on its own, and in combinations where the image was on, that effect vanished. The best combination was the new headline with the existing button text and no image.

Working out why with Mia, the problem was screen length. The new headline already carried the explanation, so the image repeated it, and the button was pushed that much further down the first screen. Measuring elements one at a time could never have reached that conclusion, because all three experiments would have answered positively. The interaction was the finding, and only this design could report it.

The most common reading is that it's a better method than a two-way comparison, since learning several things at once looks efficient. But the two have different traffic requirements. On a screen with a few hundred visitors a day, splitting into eight combinations means no combination reaches a conclusion and three weeks are gone. Checking traffic first is the order of operations, and for most teams two-way comparison is the default and this is the exception. Traffic is a constraint here, not a preference to argue about.

Reading more elements as more learning is the other one, because adding one more element looks like little extra work. But one more element doubles the combinations and halves the sample in each. Experiments with five elements usually end with all thirty-two combinations reported as inconclusive. Learning less and learning it firmly is how this method gets used. Two elements with a suspected interaction is the shape that works most often.