Dark Launch

Most checks that happen before a release happen under conditions that don't match the real thing. A test environment holds a few hundred records where production holds a few million. Neither the hours when traffic piles up nor the strange input real people type gets reproduced.

So releases looked like this for a long time. Ship the feature that passed review at three in the morning, then watch the support queue after breakfast. Roll it back if something breaks. Until the rollback lands, every user is living with the problem. Picking three in the morning was itself the mitigation: fewer people awake meant fewer people caught by whatever broke.

The awkward case was always work that leaves the screen alone and swaps out what sits behind it. Changing how data is stored, or replacing a search engine, is supposed to look like nothing happened — and whether it actually works was unknowable until real traffic hit it.

Dark launching inverts the order. It separates opening a feature to users from running the code, and creates a stage where the code runs with nothing opened. If progressive delivery is the practice of widening exposure step by step, a dark launch is the first step, where exposure is still zero.

When a real request arrives, run both the old path and the new one, return only the old path's answer to the user, and record the new path's answer for comparison. Five steps.

1. Plant the switch. A feature flag — a switch that turns a capability on and off while the code stays deployed — goes in first. Whether the new path runs at all hangs off that flag.
2. Run both paths. One incoming request goes to the old search engine and the new search engine as the same query at the same moment.
3. Return the old path's answer. What the new path produced is written to logs and never rendered. If it's wrong at this stage, nobody outside the team finds out. Teams often sample rather than keep everything, since a full record of both answers at production volume gets expensive quickly.
4. Compare the two. Count how often the answers differ, and watch the new path's response time and error rate. Set the bar before starting — something like under 0.1 percent disagreement over a full day, response time within 1.2 times the old path.
5. Turn the switch on once the bar is met. From there it moves to a canary release, which opens the feature to a small group for real, and then to a percentage rollout that raises exposure in stages.

Step 4 only works if the inside of the system is visible from outside. That property is called observability, and a team without it can run a new path for days without noticing what diverged. Logging which request produced each mismatch is what makes the cause findable.

The stage usually lasts from a few days to a couple of weeks. If load spikes at a particular moment, like end-of-month billing, passing through that moment once belongs inside the stage.

Three situations suit it.

1. Work that keeps the screen and replaces what's behind it. The old path and the new one are supposed to agree, so the answer to compare against already exists.
2. Work where load has to be measured in advance. Real request volume arrives as it is, which beats a load test built from generated traffic. Generated requests tend to be more regular than real ones.
3. Changes that are hard to undo. When a rollback is heavy — a change to how data is stored, say — a longer stage before opening anything is worth the wait.

The limits are just as clear.

1. It can't cover a feature that changes the screen. Finding out whether users press the new button means eventually showing them the new button. A dark launch has no way to tell you whether anyone would have pressed it. That work skips straight to opening for a small group.
2. Anything with an effect outside the system must not run twice. Fire both paths on a payment authorization or an email send and the user watches money leave the account twice. Those calls get stubbed in the new path so it records what it would have sent instead of sending it.
3. It costs something. Two paths running at once add server load, and branching code is harder to read. So the day the flag comes out gets written down the day it goes in. Unlike a kill switch, which shuts a feature off the moment something goes wrong, this cleanup happens after the thing has succeeded.

"Let's dark launch it for a week and decide based on the mismatch count."

Whether to open the new path should come from comparison data, not from a feeling.

"This one sends email, so we can't dark launch it."

Running it twice would have a real effect on the user's side.

"It's dark launched and the new path responds three times slower. We shouldn't turn it on."

The bar wasn't met, so it doesn't move to the next stage.

Pagedeck used this when document search moved to a new engine. Theo set every search request to go to both the old and the new engine with the same query, while only the old engine's results reached the screen. From the new engine he recorded how much the top ten overlapped and how long it took.

The first two days the overlap sat in the sixties, because the two engines split Korean document titles differently, and users felt nothing the whole time. After a settings fix and four more days, the overlap passed 97 percent and Mia called it: turn the switch on for internal accounts first.

The most common misreading is that nothing reaches users, so nothing can go wrong. Invisible sounds like harmless. But the new path reads the same database and consumes the same server capacity. Doubling request volume can slow everything down, and a badly written query on the new path can hold the database long enough to stall the old path too. A dark launch needs its own off switch, and it gets flipped the moment the numbers turn.

Treating it as a replacement for testing is the other one. Running against real data makes the earlier stages look skippable. What this stage catches, though, is only what shows up under real conditions. Checking that a known input produces a known output still belongs upstream, and skipping it fills the mismatch log with thousands of entries where nothing real can be picked out.