Change Failure Rate

Teams that are afraid of deploying tend to have a rule or two. Nothing goes out on Friday. Releases happen after 10pm. Two people stay online on release night. The rules are reasonable on their face. What nobody can say is how much risk any of them is protecting against.

Pagedeck had this argument. When someone pushed to get a new feature out faster, engineering answered that it would cause an outage, and the other side asked whether last time hadn't gone fine. Both claims rested on memory. One side remembered the single release that broke badly, the other remembered dozens that passed quietly, and the same quarter produced opposite conclusions depending on who was talking.

Change failure rate is that argument with a denominator attached. It doesn't ask how many incidents there were; it asks how many out of how many. Three incidents in a quarter sounds like a lot on its own, but it reads differently if there were 200 deployments in that quarter. Turning a feeling of danger into a ratio is what lets "ship faster" and "ship safely" meet on the same table instead of trading anecdotes.

Divide the number of deployments that ended in failure during a period by the total number of deployments in that period, then multiply by 100. Eighty releases over four weeks with six of them causing problems comes to 7.5 percent.

The calculation has three steps.

1. Settle the denominator. Count only deployments that actually reached production. Pushes to a staging server, or code shipped with the feature switched off, never reached users and stay out of the count.
2. Settle what counts as a failure. This is the part that moves around most. Write down in advance whether you count only a rollback to the previous version, whether a hotfix — a quick patch shipped immediately for one urgent problem — counts too, and whether flipping a feature switch off to contain the damage belongs in there.
3. Divide. Six over eighty is 0.075, or 7.5 percent.

Skip the agreement in step 2 and each quarter's number gets measured with a different ruler while sitting in the same table. Counting only rollbacks gives 4 percent one quarter; counting hotfixes too gives 11 percent the next. That looks like a threefold decline when the only thing that changed is the counting. So the definition gets written on one line next to the metric, and when it changes, prior quarters get recalculated on the new basis.

Whether user impact belongs in the definition is another decision. If login broke for five minutes at 4am and nobody noticed, teams split on whether that was a failure. Judging by whether a corrective action was taken usually holds steadier than judging by minutes of impact, because the record of a revert is unambiguous and the estimate of who was affected is not.

Research puts strong organizations somewhere between the single digits and 15 percent. Read on its own, though, this number is almost always read wrong, because a team that ships nothing has a change failure rate of zero percent.

That's why this metric is kept in a set of four. There's deployment frequency, how often you release; lead time for changes, the time from finished code to running in production; and time to restore service, how long it takes to get back to normal after an incident. The first two measure speed, change failure rate and time to restore measure stability. Between a team shipping eight times a week at 6 percent and a team shipping once a month at zero percent, the first one is in better shape.

The thing that actually moves this number down is usually one piece of machinery: CI/CD, the setup that verifies code changes automatically and carries them through to deployment. When checks run automatically before anything ships, the things people used to miss get caught, and when each release carries a smaller change, finding the cause after a break gets easier too.

Setting an allowance in advance is another approach. An error budget — the amount of failure you can spend while still hitting your reliability target — gets fixed up front; while budget remains, features keep shipping, and once it's spent, time moves to stabilization. It converts "what percentage is good" into "how much did we agree to absorb this quarter," which is a question a team can actually answer.

"Change failure rate is 9 percent this quarter, and half of it is in payments."

The concentration matters more than the overall figure, and the thing to fix is one area rather than the release process.

"We started counting hotfixes as failures last month. That's why the number went up."

The metric didn't get worse; the definition got wider.

"Failure rate is fine but time to restore is two hours. We can't revert."

The problem right now isn't how often things break but the lack of a way back when they do.

The first time Pagedeck measured this, it was 12 percent. Theo pulled three months of deployment records and went through everything marked as a failure one at a time: five of the eight looked identical. Each was a Friday afternoon release carrying a week's worth of accumulated changes.

So what got fixed was the sequence, not the metric. Changes started going out the day they were ready, and anything that failed its checks was blocked automatically. Two quarters later deployment frequency had gone from twice a week to eight times a week, and change failure rate had dropped to 5 percent. Smaller releases simply carry less that can break.

The most common reading is that closer to zero is better. The word failure is right there in the name, so lower looks better. But this is a ratio with a denominator, and shrinking the denominator brings it down as far as you like. Demand zero from a team and people start postponing releases and accumulating changes, which makes each release bigger and the real risk higher. A report of zero percent across a full quarter isn't a sign of a job well done; it's a prompt to ask what isn't getting shipped.

Reading it as an individual's score is common too. Whose deployment broke is right there in the record, so it's easy to tally. But deployments usually fail because of the path they traveled rather than the person who pushed them — checks that didn't run automatically, no way to revert, a change too large to ship in one piece. Attach this number to individuals and people move toward not reporting failures, quietly patching small problems without leaving a trace. From then on the number improves while the outages the team actually lives through stay exactly the same.