Observability

Back when a service ran on one server, nobody needed a word for this. Something looked wrong, you logged into that server, you opened the log file. A log is the running program's own account of itself, one line per thing it did, and reading a single file end to end took minutes.

That stopped working once there were ten servers and the one program had been split into separate pieces for documents, permissions and billing. A user presses save once and that request passes through four of those pieces in turn. When it failed, the only fact anyone had was that it failed. Which piece it died in was a mystery, and most of the time spent on an outage went into opening logs on ten machines and lining up their timestamps by hand.

So every incident played out the same way. A customer wrote in to say saving was broken, three engineers each logged into a different server, each opened the file they happened to suspect, and an hour later three versions of "looks fine on my side" came back. The whole time, the status dashboard stayed green.

That dashboard is worth looking at closely, because it explains the gap. A dashboard answers the questions someone thought of when they built it — CPU load, memory, how many servers are alive. Those are useful questions, and none of them was this outage. The failure was that the permissions piece had gone slow for one particular kind of document, and that shape of failure is different every time, which means it can never be on the list in advance.

Observability turns that around. Instead of going into a server each time something breaks, you arrange in advance for the system to emit enough signal on its own. When the signal is rich enough, an unfamiliar failure can be narrowed down from material that is already outside the system, without logging into anything. Because the word describes how much you have rather than a thing you install, it isn't on or off. What's measured is how wide a range of questions you can answer without going back to the source.

Three branches of engineering deal with a service while it runs. Observability plants the means to see inside. Incident management is the agreed procedure for who moves first and what gets restored when something actually stops. DORA metrics score how often and how safely the organization ships, using four values such as deploy frequency and time to restore. Observability comes first of the three. With no signal planted, incident management gets its starting gun from a customer email, and any attempt to measure recovery time runs into the problem that nobody can say when the outage began.

The terms under this one fall into three branches: the signals a system emits, the tools that collect and display them, and the reliability targets that decide which readings count as fine. In that order, you produce signal, read it, and judge it.

Signals are the raw output, and there are four. The first three answer different questions and the fourth carries the answer to a person.

1. Logging — a written line for each thing that happened. "14:02, save request from user 8821 failed on timeout." This carries the most detail about any single event, but millions of lines pile up in a day, so it is worthless unless you already know roughly where to look.
2. Metrics — counts of the same quantity stacked along a time axis: requests per minute, failure rate, response time. Metrics will tell you to the second that the failure rate jumped from 0.2 percent to 14 percent at 14:01. They will not tell you why.
3. Distributed tracing — the path one request took through the pieces, stitched together in order. A single view shows that the save went through the document piece, sat in the permissions piece for 1.8 seconds, then reached storage. Once there are several pieces, nothing else turns "which one is slow" into a fact instead of a guess.
4. Alerting — the mechanism that contacts whoever is on duty when a condition is crossed. The first three are only useful if somebody is watching, and at three in the morning nobody is.

The first three aren't alternatives to each other, they divide the work. Metrics own "since when," tracing owns "which piece," logging owns "because of what." Plant only one and you stall at the same step every time. With just a failure-rate graph you learn that it started at 14:01, and then you log into a server. With only logs, you don't find out there is a problem until a customer tells you. They also differ in what they cost to set up. Logging and metrics start with a few lines of code; tracing needs the same marker attached in every piece and carried across the boundaries, which is more work. Most teams lay down metrics and logging first and add tracing when the pieces have multiplied enough that tracking down a cause has become the slow part.

Tools are the second branch, and there are two. Application performance monitoring (APM) shows which screens and which requests are slow, organized around response time. Error tracking collects thrown exceptions and groups identical ones, so you get "this error, 4,200 times today, affecting 310 users" instead of 4,200 separate lines. Read raw, those lines look like 4,200 problems; grouped, it becomes clear there is one thing to fix. The two overlap enough that a single product often does both. What a tool covers has an edge, though. It will find the slow request on its own, but which step inside that request is slow shows up only as far as someone planted markers.

Reliability targets are the third branch, where the readings get turned into a line you either hold or don't. Four of the five form a chain, and the fifth is the oldest form of the same idea.

1. Service level indicator (SLI) — the value actually measured. Save success rate, or response time at the slowest one percent of requests, picked to sit as close as possible to what users experience.
2. Service level objective (SLO) — the level the team commits to holding, written as one line such as 99.9 percent save success. Without this line there is no threshold to hang an alert on.
3. Error budget — the amount of failure you can spend and still meet the objective. A 99.9 percent target leaves 0.1 percent to spend. While budget remains, ship new work aggressively; once it's gone, stabilize before shipping anything else.
4. SLA, the service level agreement, is the same promise written into a customer contract, usually with money back attached if it's missed. Because the penalty is real, the contracted number is normally set looser than the internal objective.

Uptime is the oldest of these: the share of time the service was working. A month at 99.9 percent allows 43 minutes down. The problem is that a service which never stopped but answered every request in eight seconds counts as fully up by this measure. That gap is why teams have moved from a single uptime figure toward an indicator and an objective written together.

The three branches run one way. Signals produce the raw material, tools make it readable, targets decide whether the current reading is acceptable. Miss the front and the back cannot stand: write a response-time objective without planting tracing and, on the day you miss it, there is no way to know what to fix. The reverse gap hurts too. However finely you plant signal, with no agreed line for normal you end up staring at a response time that grew from 0.4 to 1.1 seconds and deciding in a meeting, every single time, whether that is today's problem or next week's.

"The logs don't tell me which step slowed down. Is this request traced?"

The record exists, but the path the request took was never captured, so the cause can't be narrowed.

"If it pages forty times a day nobody reads it. Let's fire only when we're past the objective."

The firing condition is loose enough that real outages are getting buried, so the threshold needs rebuilding.

"We've spent 60 percent of the budget this month. Let's roll the next release out in stages rather than all at once."

Little room for failure is left, so ship in a way that lowers the risk.

Pagedeck only started this work after that Saturday outage. The first thing Theo did was not to pick a tool but to write down the questions: is saving failing right now, is it everyone or some people, which piece is it dying in, and since when. Writing out which signal each question needed settled what to plant — success rate and duration of save requests counted as numbers, an identifiable user and document attached to every failed request in the log, and one marker carried across pieces so the path could be stitched back together.

Two months later the same symptom returned and one screen ended it. The failure-rate graph bent at 11:12 that night, the trace view showed the permissions piece holding each response for three seconds, and that piece's log repeated a line about the connection count hitting its ceiling. Theo did log into a server, but to fix it, not to find it. Mia pointed out something else in the review: last time Sofia's note to the enterprise customers went out after recovery, and this time it went out before.

The most common mistake is believing that buying a tool gives you observability. Installation fills a screen immediately and the graphs look convincing, so the belief is easy to hold. But what a tool gathers by itself is the signal that every service has in common. Whether this team's save flow died at the permission check or at storage is something a person has to leave behind in that code. Skip the distinction and you sign a contract worth millions of won a month and still find yourself, mid-outage, looking at a CPU graph.

The other one is assuming more is better. Keep everything in case it's needed later sounds like the careful choice. Two things break at once. The first is cost, which scales with volume for both storage and search, so unread logs outgrowing the server bill is an ordinary occurrence rather than a rare one. The second is that signal drowns in noise. Record the successful path as well and the one decisive line from the moment of failure sits among 30,000 lines from the same second, which puts the time to find it back where it was when you recorded nothing. What to keep gets decided backward, from the list of questions. Write the questions you have to answer first, keep only the signal those questions need, and the volume drops while the screen you look at during an outage gets sharper.