Preference Test

When the options split two ways, the decision often failed to happen in the room. The evidence on each side was somebody's instinct, so whoever talked longest won.

Pagedeck's pricing screen went that way. Noa had two versions on the table; one side argued the price looked too large, the other argued the price was buried and therefore untrustworthy. Both arguments were claims about what a first-time viewer would see, and nobody in that room was a first-time viewer. People who had looked at the screen a dozen times were guessing at the reaction of people who hadn't.

A preference test moves that guess outside the building. Several dozen people who have never seen the screen get both versions together and pick one, under the same conditions and in a randomised order. What comes back isn't a correct answer, it's the viewpoint the room was missing. Ask for the reasoning too, and the sentences people write are often worth more than the split itself.

The first step is deciding in one sentence what you're asking about. Putting the options side by side comes after that.

1. Narrow the question to one attribute. Ask which one is better and you get taste; ask which one looks more trustworthy and you get judgement. For a pricing screen, the attribute might be which version makes the price easier to understand. Write that sentence down before the options are exported, because the attribute you pick also decides what the two versions should differ on.
2. Keep it to two or three options. Past four, responses scatter and no difference survives. Show four versions to 40 people and you get something like 28, 26, 24 and 22 percent, which supports no conclusion at all.
3. Hold everything but one dimension constant. Two versions that differ in colour, layout and copy at once can win without telling you what won. To compare a table layout against a card layout, keep the wording and the colours identical on both.
4. Randomise the order per participant. Whatever people see first becomes their reference point, so a fixed order quietly favours the version in front.
5. Collect the reasoning as an open question right after the choice. One or two sentences about what they looked at is enough. That text is what tells you where to work next, and it's the part of the result that survives the project — a comment saying the plan names were unclear applies to every future version of the screen, while the split applies only to the two you tested.
6. Recruit 30 to 50 people who could plausibly use the product. This study reads proportions, and five people can't distinguish a 60/40 split from an even one.
7. Read the split and the reasons together. Something like 62/38 supports picking a side. At 52/48, treat the two versions as equivalent and act only on the points the comments raise in common.

It fits best when direction is deadlocked and there is nothing shipped to measure. Before launch there's no behavioural data, so an impression collected outside the building beats an argument inside it. It also suits work like logos or illustration, where no performance metric settles anything, and it's cheap enough to run in a day, which matters when the alternative is a second week of the same meeting.

Four limits are worth knowing.

1. Preference doesn't predict performance. The version people say looks better can differ from the version people actually click. If the screen can be shipped, an A/B test — splitting traffic between two versions at random and comparing outcomes — answers the question more accurately.
2. Participants aren't designers. "Blue would be better" is not a design instruction. What that answer is worth is the fact that the current screen reads as cold, not the colour. Treat every answer as a symptom and keep the diagnosis on your side of the table.
3. It says nothing about usability. People are choosing from a picture, so whether they can complete a task belongs to a usability test.
4. Skip the reasoning and it ends as a taste poll. A 60/40 number by itself leaves nothing to fix.

The method mixes quantitative research, which measures proportions, and qualitative research, which captures reasons in words, inside a single question. That makes it shallow on both counts and valuable early, while the field of options is still being narrowed.

"We ran a preference test on the two versions and it came back 62/38."

One version pulled clearly ahead, so the recommendation is to take it.

"51/49 means no difference. Let's read the comments."

The proportion settled nothing, so look for what the written answers have in common.

"Skip the preference test and A/B it. That screen is already live."

Behavioural data is available here, so the impression study isn't needed.

Noa ran this on two versions of the pricing screen. Of 44 participants, 27 chose the table layout, but the comments showed that half of those 27 hadn't found the annual discount line. The decision went to the table layout, and the same meeting created a task to move the discount line. The split decided one thing — direction — and the written reasons decided everything else.

The most common misreading is that the version most people chose will also perform better. It looks reasonable because real people made the choice. But respondents are comparing two screens next to each other, not deciding whether to pay for something. Real users see one screen, with no alternative to compare it against and no researcher waiting on an answer. Shipping a preference result as if it were a performance result, watching conversion drop and rolling back does happen, and in that case the study wasn't wrong — it was answering a question nobody asked it.

The other habit is substituting an internal vote. Everyone looks at the same screens and picks, so the result should be similar. In practice the team has seen the screen dozens of times and no longer knows where a newcomer stalls. An abbreviated menu label reads fine internally and not at all outside. Decide by internal vote and that gap ships with the screen, then arrives back as support tickets. There's a second problem too: if the person who made one of the versions is in the room, which one they prefer travels to the people sitting next to them.