Root Cause Analysis

Putting out the fire and preventing the next one are separate jobs. By the time recovery is done, though, everyone is tired and other work is waiting, so incidents routinely close without anyone deciding what needs to change.

Pagedeck had a save outage that went that way. Theo restarted the server, recovery took 20 minutes, and the record said "out of memory." Two weeks later the same outage happened again and took an hour. What the first record captured wasn't a cause, it was the last stage of the symptom. Why memory ran out, and why that situation arose for the first time that day, went unasked.

The second occurrence of an incident costs more than twice the first. Users lose confidence on the repeat, and the team spends people on the same outage twice. Treating cause analysis as work distinct from recovery exists for that reason.

Build a timeline first, then start at the symptom and keep asking why until you reach a condition you can change.

1. Write the timeline as facts only. Lay out what happened at what minute, and put deployments, configuration changes and traffic shifts on a parallel line at the same timestamps. Candidate causes usually appear where those two lines meet, and disagreements about the sequence surface here rather than halfway through the discussion.
2. Chain the whys from the symptom. At Pagedeck the chain ran: saves failed, the server ran out of memory, one request loaded an entire document into memory, document size had no ceiling, and no ceiling item existed on the design review checklist. Five links was enough to reach something the team could act on.
3. Decide where to stop. Stop when you reach a condition you can change. In that chain, adding the item to the checklist and setting a document size ceiling are those conditions. "The engineer didn't check" is somewhere to keep digging rather than somewhere to stop: a structure that lets an unchecked change through is the condition.
4. Don't narrow to a single cause. Incidents usually need several conditions to coincide, which is also why the same system can run for months with one of the holes already open. If there was no size ceiling, the alert threshold sat above the real limit, and a customer happened to upload a 300-page document that day, write down all three. Fix one and the next occurrence arrives wearing a different shape.
5. Give every prevention item an owner and a date. Items written as prose are still there six months later. "Document size ceiling, Theo, this week" and "alert threshold adjustment, next week" go into the actual backlog in that form.
6. Verify that it's blocked. Recreate the conditions in a test environment and check that it fails this time — upload a 300-page document and see the ceiling reject it. An unverified fix is a record of a fix.

Use it on any incident with wide impact, and on anything that has happened twice. Repetition is the signal, because it means the first analysis stopped at the symptom and the condition underneath it is still in place. An incident whose cause is obvious from the start — one configuration line changed and reverted in three minutes — doesn't need this run separately.

There are four limits.

1. It can't happen while the fire is burning. Recovery comes first and this comes after; attempting both at once delays the recovery. Keeping notes during the recovery does make the next day's timeline much easier to build.
2. It stops producing facts the moment it becomes about blame. On a team where who pressed what comes back as an accusation, records get written carefully, and carefully written records don't reveal structure. The posture has its own name, blameless postmortem.
3. The premise of one cause is often wrong. The more layers a system has, the more an incident is several defences failing together, and naming one of them the root cause leaves the other holes open for the next time the conditions line up.
4. Analysis doesn't produce outcomes. What prevents recurrence is the items from step five actually being done. Without tracking completion of those items, what accumulates is analysis documents, and the next incident of the same kind arrives with a write-up of the last one already on file.

"Out of memory is the symptom, not the cause. Why it ran out is still open."

Asking for one more level of digging.

"The same outage recurred in two weeks, so last time's analysis stopped at the symptom."

Reading repetition as evidence the analysis was shallow.

"Put an owner and a date on each prevention item."

Filtering out the items that were never going to happen — and making sure they don't all land on whoever is on call.

After the second outage, Theo and Mia spent ninety minutes building the timeline and chaining five whys. Three items came out: a ceiling on document size, lowering the memory alert to 70 percent of the real limit, and adding a size-ceiling item to the design review checklist. All three went into the backlog with an owner and a date and were finished within three weeks. That outage hasn't recurred; two support tickets did arrive from customers hitting the ceiling, and the ceiling value was adjusted once.

The most common reading is that this is a hunt for one cause. The name contains a singular, so the assumption is natural. But incidents rarely have a single condition. No ceiling, a late alert, and a large document arriving that day all had to coincide. Pin one as the root cause and close the incident, and the other two holes stay open, so the next occurrence arrives in a different order and looks like a new problem.

The other belief is that producing the document is the point. There's a template, a meeting happens, a file gets saved, and it feels finished, especially when the write-up reads well. What prevents recurrence is not the document but the items that came out of it. If an item has no owner, no date, or no place in the backlog, that analysis was two hours spent changing nothing. Measure this work by whether the same incident happened again, not by how many write-ups exist.