Novelty Effect

Numbers jumping after a redesign and settling back a few weeks later is an old pattern. What was missing was a name for it, so teams usually filed it as "it worked at first and then cooled off" and went looking for the cause somewhere else — seasonality, a smaller ad budget, an operations team that had gotten sloppy.

Once A/B testing became routine, the illusion started showing up more often and costing more. An A/B test splits users at random between two versions and compares the results, and because the comparison is built in, it looks like a winner can be called within days. Those first days are exactly the window where the distortion is largest.

Pagedeck learned this the expensive way. Noa rebuilt the sidebar and ran it as an experiment; three days in, document creation was up 14 percent, and the team shipped it to everyone on the spot. A month later, document creation was back where it had been before the change. Digging through the logs, Theo found that most of the first week's increase was existing users clicking the rearranged menu once to see what had moved. Novelty effect is the name for that few-day spike.

It comes from existing users noticing that something changed and pressing it once to find out what it does, all of which lands at the front of the experiment. They're not using the new thing because it's better; they're checking. Once the checking is done, the behavior stops.

That's why new users barely produce it. For someone opening the product for the first time, the new screen is just the screen, with no earlier version to compare against. The fact that novelty shows up only in the existing-user population turns out to be the key to detecting it later.

In the data it decays. A 20 percent lift on day one becomes 12 percent by day three, 4 percent in week one, and roughly nothing by week two. Average the whole period into a single number and that early hill goes into the average intact, so even a two-week test reads inflated.

The same thing happens in reverse often enough to be worth knowing. Long-time users whose familiar controls moved will fumble for a few days and the metric drops, then recovers once their hands relearn the layout. Either way, the first few days are not trustworthy in either direction.

There are three ways to strip it out.

1. Look at new users on their own. If there's no difference in that group, the lift you saw was probably exploratory clicking.
2. Break the period into weeks. Fourteen percent in week one and 2 percent in week three means the remaining 2 percent is closer to the real effect.
3. Exclude the first few days from the readout, deciding how many days to drop before the experiment starts. Decide afterwards and you'll pick whichever window flatters the result.

The first two work best together. New-user numbers give a baseline with no novelty in it, and the week-by-week numbers show how much novelty has drained out of the existing-user side. When the two converge on a similar figure, treat that figure as the effect. When they don't, something other than novelty is going on and the experiment needs a closer look before anyone ships anything.

It comes up first when deciding how long to run a test. Anything that changes the interface runs at least two weeks and in whole-week blocks, and this is the reason. Usage differs by day of the week, so periods are set in multiples of seven. Three other places come up regularly.

1. Calling the result. Instead of one averaged gap against the control group — the group left untouched as a basis for comparison — the gap gets laid out week by week. A difference that shrinks every week is a reason to hold off on declaring a winner, and a difference that holds steady is the strongest evidence a short test can produce.
2. Reading metrics after a launch. Usage in the week following a release is almost always high. Pairing that with retention, the degree to which existing users keep coming back, shows whether any usage survives once the excitement drains.
3. Writing up the experiment log. Recording the week-by-week numbers alongside the verdict keeps whoever revisits this area in six months from reading the first-week figure and reaching the same conclusion again.

"We can't call it on three days of data. That could be novelty — let's run it two weeks minimum."

The difference on screen may not be real yet, and the ask is for more time.

"Split out new users and the difference basically disappears. I think that was novelty on the existing side."

A verdict that the lift was a reaction to newness rather than to the change itself.

"Week one was 14 percent, week three is 2 percent. Two percent is the number with novelty taken out."

Pointing at what remains after the inflated part is removed.

Since the sidebar episode this check has been part of the process at Pagedeck. Mia set a rule that interface experiments aren't called before two weeks, and Theo added a panel to the results screen showing the gap for each week side by side. When Noa tested the next redesign, week one came in at 9 percent and week three at 6 percent, and the team shipped on the strength of the 6.

Watching the number fall isn't the disappointing part it sounds like. The bar simply moved from the first week to the third, and in exchange the team stopped watching metrics collapse after a launch.

The most common conclusion is that a change showing novelty was worthless. Hearing that the lift was an illusion gets taken as hearing that all of it was. But novelty doesn't erase the real effect; it stacks on top of it. If 12 of week one's 14 percent was exploratory clicking, the remaining 2 percent is still there. What's left after peeling off the added layer is the number to judge, and throwing the whole thing out means repeatedly discarding small improvements that would have compounded.

The second mistake is assuming a longer test fixes it by itself. "We ran it for two weeks" gets used as a synonym for safe. Average those two weeks into one figure, though, and the first three days' hill sits inside the average exactly as before. Running longer supplies the material to strip novelty out; the stripping is done by whoever breaks the result down by week. Skip that step and a two-week experiment produces the same conclusion a one-week experiment would have.