Smoke Test

The moment right after a new version goes out is the most dangerous one. If something is broken, every user is hitting it from that second onward. But running the full test suite takes time. A collection of 500 end-to-end tests — tests that launch a real browser and click through what a user would do — takes about two hours for one pass.

During those two hours, users are on the new version. A team that deploys on Friday afternoon and starts the full run finds out 40 minutes later that login is broken, by which point a dozen support tickets are already waiting.

The opposite choice was no better. Finish the deploy without checking anything and nobody knows until somebody happens to open a screen. For a deploy at midnight, that means the following morning, by which point the first users of the day have already met it.

What was needed was not an instrument that answers whether everything is correct, but one that answers whether to roll back right now, within two minutes. The name comes from hardware: powering up a newly built board to see whether smoke comes out. If it does, the remaining checks are pointless and you stop there.

Pick only the flows that a rollback decision depends on, run them automatically right after deploying, and attach the response to the result.

1. Choose the targets. One criterion: include only flows whose failure makes the service effectively absent. Login, first screen render, opening a document, saving a document, reaching the payment screen — five to ten of them. Things you could learn about a day late, like search result ordering or notification preferences, stay out.
2. Set a time ceiling. The whole run should finish inside two to three minutes. Fix the ceiling first and cut items until they fit; doing it the other way around lets the list grow forever. Two or three minutes is roughly as long as somebody will stand and watch a terminal before moving on.
3. Run against the environment you actually deployed. Do not substitute anything. Use a mock to fake the payment provider's response and you will miss exactly the case where this deploy's configuration broke the connection to that provider.
4. Attach a verdict and an action. A single failure either triggers an automatic rollback or pages whoever deployed. If a human has to go look at a results page, the two-minute goal is already gone.
5. Clean up what it leaves behind. Running in production means the accounts and documents the test creates land in real data. Keep dedicated accounts and build the deletion step alongside. Test rows that look like customer rows become their own small incident the first time someone exports a user list.

With those five in place it becomes the last cell of the deploy procedure: push, wait two minutes, walk away on green, roll back on red.

Four situations suit it: immediately after a deploy, after standing up a new environment, when an external service is suspected of being down, and on a schedule every 30 minutes with no deploy involved. That last case catches things stopping when nothing on your side changed. It confirms the flow itself is broken before an observability tool reports it in aggregate numbers.

Four limits are worth knowing.

1. Passing does not mean healthy. Five flows were checked and the rest were not. Login can work and documents can open while the export feature is entirely broken, and the light still comes up green.
2. Failing does not locate the cause. It tells you login is down, not where it broke. That comes from unit tests, integration tests, and the logs.
3. It leaves traces in production data. Skip the cleanup step and six months later the user list holds 400 test accounts, with usage metrics inflated by the same amount.
4. Add enough items and it stops being a smoke test. A smoke test that takes fifteen minutes is one in name only, and what it actually does is delay the rollback decision by fifteen minutes.

"Let's just run smoke and go home. The full suite is scheduled overnight."

Do only the checking a rollback decision needs, now.

"Should we add document sharing to smoke? We're at two and a half minutes and this adds thirty seconds."

Weighing the item and the time ceiling together.

"Smoke passed but export is broken. We weren't watching that far."

Confirming that passing does not mean everything is fine.

At Pagedeck, Theo wired six checks into the end of the deploy pipeline: login, document list, opening a document, saving a document, following a share link, and rendering the payment screen. The whole thing finishes inside 100 seconds, and a single failure rolls back to the previous version automatically and sends an alert. The payment screen was not on the original list; it went in after a deploy shipped a bad payment provider setting that nobody noticed until the next morning.

The suggestion to add more items comes up every quarter. Mia asks for each new feature to be included, and Theo shows the ceiling first. What does get added goes into the regression test suite that runs overnight instead. The six checks live in their own file, so adding one shows up as a visible change in review rather than quietly.

The most common error is reading a passing smoke test as a successful deploy. It is the last green light on the deploy screen, which invites that reading. But what the green light reports is that five core flows are alive. A broken feature outside those five is not caught here and usually arrives as a support ticket the next day. Read the light as deploy success and you will not connect that ticket to the deploy at all, which is precisely the day the distinction matters. The sentence this result supports is not "the deploy worked" but "there is no reason to roll back right now."

The second error is assuming more items is better. Checking more things does not sound like it could hurt. But the value of this test comes from how long it takes, not from how many items it has. At two minutes the person who deployed watches the screen and waits; at fifteen they go do something else and see the red light twenty minutes later. Those twenty minutes erase the reason the check exists. When there is more to verify, it belongs in the suite that is not racing a clock.