Flaky Test

Automated tests are worth having because their results keep a promise. Green means nothing you didn't touch is broken. Red means something you just changed broke it. That promise is what gives an engineer the nerve to edit code somebody else wrote.

The scene where the promise breaks always looks the same. At Pagedeck, Theo pushed a change to how documents sort and the run came back red. The failing test covered the checkout screen, nowhere near anything he had edited. He spent thirty minutes looking for a connection, then re-ran it on a hunch and got green. He merged.

What came next is the actual problem. Within a couple of weeks the whole team had learned to hit re-run before opening the log. Then one day a genuinely broken change rode that habit straight through. That's why flaky tests get a name of their own. It isn't one failing test — it's the signaling system going bad — so teams track it separately from ordinary bugs.

Flakiness appears when a test depends on something other than the code: a condition that varies from run to run. The causes fall into four groups.

1. Depending on time. Wait two seconds for a screen to appear and then move to the next line, and the test fails on a day when the server takes 2.1 seconds. This is most common in end-to-end (E2E) tests, which drive a real browser through the steps a user would take.
2. Depending on order. One test uses the account an earlier test created. Everything holds while they run one after another, and falls apart the moment the suite runs several at once to save time.
3. Depending on the outside. Send real requests to somebody else's server and their situation becomes your result. A mock — a stand-in that simply answers "success" — exists to cut that dependency out; skip it and point the suite at a payment provider's sandbox, and every maintenance window on their side turns the build red.
4. Depending on environment and randomness. A test that reads the current clock breaks only across midnight or at month end. A test built on random values breaks only when it happens to draw an edge case. Time zones and locale settings differing between a laptop and the build server produce the same effect.

Finding them is straightforward. Run the same test 50 times without touching the code and count the failures. One failure means it isn't deterministic; above a three percent failure rate it goes on the list. That number settles arguments far faster than someone saying a test "seems weird lately." It also tells you where to start: a test failing once in fifty is usually a timing assumption, while one failing a third of the time is usually shared state between tests.

When a fix isn't available right away, pull the test aside — out of the blocking run, into a separate job, with an owner and a date attached. Let that quarantine list grow unchecked, though, and it amounts to deleting the tests.

The term comes up most often around continuous integration (CI), the practice of merging changes frequently and verifying each one automatically. CI decides whether a change can merge on a single red or green signal, so an unreliable signal undermines the practice itself. On a team whose checks take 15 minutes, half of them false alarms, hours a day go into waiting for nothing.

It also shapes how the test layers get balanced. A unit test, which isolates a single function, depends on almost nothing external and is rarely flaky. E2E tests carry the browser, the network, and the clock all at once, which is where most instability collects. The usual advice to keep E2E coverage down to a few dozen flows like signup and checkout rests on this as much as on speed.

Retry policy is another place it surfaces. Automatically rerunning a failed test quiets things down immediately, but it hides the symptom while leaving the cause. Teams that do use automatic retries usually count how many runs were saved by one, so the hidden number stays visible.

Keeping a record of which tests have flaked, and how often, matters as much as the weekly number. Without one, the same test gets rediscovered every few months by whoever happens to be on call, and each person pays the thirty minutes of investigation from scratch.

That's why many teams watch it weekly. Pagedeck reviews last week's false-red rate alongside the size of the quarantine list every Monday. In a week where the rate passes five percent, that work goes ahead of new features.

"That one's flaky. Just re-run it."

The failure isn't from the change you just pushed, so don't dig through the log.

"We've been saying that for two months. Let's find the cause this time."

A test has been getting waved through with re-runs long enough.

"The quarantine list is over thirty now."

Enough tests have been pulled out of the blocking run that real coverage has shrunk.

When the Pagedeck checkout test had been wobbling for ten days, Theo ran that single test 100 times. All eleven failures clustered around 9 a.m. The payment provider's sandbox restarts at that hour.

There were two ways to fix it: replace the sandbox with a stand-in that always answers success, or wait briefly and retry once on failure. Theo took the first. The second only makes the test quiet, leaving whatever a real user would hit in that same situation exactly as it was.

The most common position is that it doesn't matter much, since a re-run passes. One incident is thirty extra seconds, so it feels that way. But the cost doesn't come out of time, it comes out of trust. Someone who has been burned by a few false reds stops reading logs, and that habit applies just as evenly to real failures. The entire reason for having tests quietly disappears while every dashboard still looks fine.

The other is treating it as a test-side concern. The unstable code is test code, so the product seems unaffected. But the fact that results shift with order or timing is sometimes a signal that the same thing can happen in the product. Defects that only appear when two requests arrive at once are hard to reproduce deliberately, and a flaky test is occasionally one of the few places they do reproduce. Delete it without looking at the cause and that window closes too.