Canary Release

Releases used to be on or off. Push the new version on Friday night and from that moment every user is running the new code. The only thing backing that decision was that nothing had gone wrong in the test environment.

Some defects don't surface in a test environment, though. They need conditions only real users create: an account made in 2015 whose settings are stored in a format the new code can't read, or one team with 8,000 documents piled up that turns the list screen into a 30-second wait.

When a problem surfaces after it's already open to everyone, the response is always late. Engineering notices around the time support tickets start piling up, finds the cause, and ships another deploy to undo it. In the meantime a few thousand people saw a broken screen, and some of them don't come back. The size of an incident is set less by how severe the defect is than by how many people were exposed to it — and an all-at-once release starts with that number at its maximum.

A canary release puts that number under your control. The new code opens to a very small group, and if something looks wrong there, it gets switched off before anyone else experiences it. The name comes from the caged canaries that detected toxic gas in coal mines first: a few take the hit so the rest are protected.

Deploy the feature behind a switch that's turned off, turn it on for a very small group, and use a pre-agreed set of measurements to decide whether to widen or close.

1. Put the work behind a feature flag and deploy with the flag off. A feature flag is a switch that turns a feature on and off while the code sits deployed. Without it, both narrowing the audience and undoing the change would require another deploy, and the rest of the procedure falls apart.
2. Decide who gets it first. Internal accounts go at the front, because those people will tell you directly when something breaks. One percent of all users comes next. Large contracts and customers who can't tolerate downtime stay out of this stage entirely.
3. Write down what you'll watch before you turn anything on. Error rate, response time, the success rate of the feature itself — pick the measurements and their baselines in advance. Choose them after looking at the numbers and you'll almost always read the numbers as fine.
4. Turn it on and watch for a set period, usually a few hours to a day. Compare the group with the feature on against the group with it off. Looking only at the treated group leaves you unable to tell a slow hour of the day apart from slow new code. How much you can tell about what's happening inside a system from the outside is called observability, and a team without it can sit at one percent and never notice things getting worse.
5. Make the call. If nothing moved, go on to percentage rollout, which raises the exposed share in steps. If the numbers got worse, hit the kill switch and turn the feature off. No deploy is involved, so it takes seconds.

Canary releases fit when the defects you're afraid of only appear on real user data. Old accounts, unusually large teams, orders of operations nobody anticipated — none of that can be manufactured in a test environment. They also fit changes that are sensitive to load, since how hard a new query hits the database is only measurable under real traffic.

Rare defects, on the other hand, slip straight through. If one percent is 300 people, an error that hits one user in a thousand will probably not appear once in that group. Watch for a day, conclude it's fine, open it to everyone, and 300 reports arrive at once. Passing a canary doesn't mean the change is safe; it means nothing large went wrong.

Results also mislead when the group doesn't represent the whole. Internal accounts use the product in a narrow way, so stopping there hides what actual customers will hit. Migrations are the other hard case. If the change also writes data in a new format, flipping the switch off doesn't remove the records already written that way, so you have to decide in advance how the old code will read them.

"Let's canary it to internal accounts only and look again tomorrow morning."

Hold off on the full release and collect a day of numbers from a small group first.

"Error rate jumped from 0.3 to 1.8 in the canary, so we turned it off."

A small group surfaced a problem, and the rollout was reversed instead of widened.

"Keep that account out of the canary group."

A customer who can't absorb an incident should be excluded from the first wave.

When Pagedeck changed how documents are stored, Theo ran the new save path behind the scenes with no visible change first, then turned the feature on for 45 internal accounts. Half a day passed with nothing happening, so he widened it to one percent of users — and that's when save failures started showing up. The error only occurred in documents holding more than 40 images, and no internal account had a document like that.

Mia pulled the kill switch and used the retro to change how canary groups get chosen. Start with internal accounts, but always pass through one percent of real customers, and check that the group includes teams with heavy documents before moving on.

The most common mistake is reading a canary release as an experiment. The shape is the same — some people see the new thing, then you compare — so the confusion is understandable. What each one asks is different. An experiment asks whether the new version performs better than the old one; a canary asks whether the new code is broken. That's why a canary ignores conversion and watches error rates and latency. Mixing the two slows the decision down: you end up leaving an obvious bug running for days while waiting for a performance metric to become conclusive.

The other mistake is treating a clean canary as the finish line. It's the opening stage. Load problems invisible at one percent routinely appear first at 25 percent, so every step up the ladder calls for the same measurements again. Stop here and open the gates to everyone and the canary stage bought you nothing.