Unmoderated Test

The standard way to find out whether a screen works was to bring someone in and sit with them. In a moderated test a facilitator hands over each task and, the moment a participant's hand stops, asks what they were looking for. Getting the reasons makes the results the richest available.

The cost is an hour per participant, and that hour has to land where both the participant's calendar and the facilitator's calendar are free. Finding five users who match the criteria and scheduling them takes two weeks on its own. Add writing up the observations and a round runs three weeks.

For a team shipping screen changes every two weeks, that cycle doesn't line up. Noa was preparing a test of the invite flow, and while recruitment was still going the screen changed twice; by the time the findings landed, the button that got criticized was gone. The test wasn't wrong, it was late.

Tools that record a screen and capture a voice track remotely made a different sequence possible. Write the task wording in advance and let participants work through it alone whenever suits them. Nobody is there to ask follow-up questions, but for the same reason ten people can run in a single day.

It starts with writing every task down in full. There is no chance to clarify anything in the moment, so the wording you wrote is all the participant gets.

1. Write what you want to know as one sentence. "Evaluate the invite screen" is not a question. "Does a user who doesn't know the invitee's email address finish an invitation?" is.
2. Write the tasks as sentences. Reusing words that appear on the screen hands over the answer, so avoid them. Not "click the invite button" but "get one of your teammates to see this document with you."
3. Recruit. Pick people who match the criteria out of your existing user list, or buy them from a panel service that brokers participants. A panel returns ten people within a day of setting the filters, but they are people seeing the product for the first time.
4. Pilot with one person. Hand the same tasks to a colleague at the next desk and see whether the wording alone makes clear what to do. Any ambiguity that slips through here wastes all ten participants' recordings.
5. Run it. Ask for a screen recording plus thinking out loud, and put a time limit on each task. Without a limit, a stuck participant wanders for twelve minutes before giving up, and those twelve minutes are a burden on whoever watches too.
6. Read it. For each participant, note whether the task succeeded and the timestamp where they stalled, then count whether the same spot recurs. One person stalling may be about that person; four out of five stalling in the same place is the screen.

Skipping the last step is common. A pile of recordings is not the same thing as a result. Share ten videos with the team and nobody opens them. What should remain is a list of things to fix, not footage.

It fits when the task is unambiguous and the screen already exists. For flows with a clear start and end — signup, invitation, checkout — counting how many people reach the end is cheapest here. It also works at the prototype stage. A prototype is a temporary build made to see reactions before the real thing exists.

It is also the right choice when you need numbers. Claiming a 40 percent success rate takes thirty participants rather than five, and comparing which of two designs stalls fewer people takes that many on each side.

Every limitation traces back to one thing: you can't ask a follow-up question.

1. Reasons stay guesses. The record shows that someone stared at the same screen for three minutes; it doesn't show what they were hunting for. Ask for thinking out loud and half of them still work in silence.
2. A misread task runs to the end. Someone who read the wording differently does a different job, and nothing corrects it. That recording drops out of the analysis.
3. Anything off-screen is invisible. You read a completion time without knowing whether they opened another window halfway through or somebody interrupted them.
4. First-time reactions get mixed in. Panel participants don't know the product and carry none of the habits a real user built up over months.

That makes it a poor fit for exploratory work. A question like why people aren't using a feature, where you don't yet know the shape of the answer, belongs somewhere you can ask follow-ups. Teams often run the two methods in sequence: go unmoderated first to count where people stall, then pick the single worst spot and bring people in to ask why.

"Let's go unmoderated with ten people this round and just look at the success rate first."

Reasons come later; right now the point is counting how many finish.

"The task wording can't contain the word 'invite'. That's the name of the button they're supposed to find."

The task is handing over the answer, which makes the result meaningless.

"Three of the ten recordings show people who understood the task differently. I'll leave those three out of the count."

Removing the misreads that nobody was there to correct.

Pagedeck ran this before reworking the invite flow. Noa wrote two tasks and pulled twelve participants from a panel: invite one teammate to a document, then change that person's permission to view-only.

Eleven of twelve finished the first task and four finished the second. All eight who failed stalled in the same place. The menu for changing permissions sat behind a small arrow next to the person's name, and participants kept searching the settings area at the top. Why they went to settings first was nowhere in the recordings, so Noa booked three moderated sessions carrying that one question.

The most common reading is that a cheap, fast method replaces bringing people in. A round costs a fraction as much and finishes in a day, so it looks that way. But the two answer different questions. Running unmoderated tells you where and how many people stall; sitting with someone tells you why they stalled. Fix a problem knowing only the where and you usually end up enlarging the element or changing its color, and if the cause was a badly chosen label, the same number of people stall in the same place next round.

Assuming more participants make the result better is also frequent. Confidence rising with sample size applies to figures like a success rate. When the goal is locating where people stall, the same problems start repeating around six or seven participants, and running thirty means confirming known problems thirty times while the watching time grows. Before adding participants, look again at the task wording. Get that wrong and a hundred recordings hold nothing usable.