Error Tracking

An exception is the signal a program leaves behind when it hits something it can't handle and stops what it was doing. The user sees one line — "something went wrong" — while the server side records which line of code gave up and why. The awkward part used to be that somebody had to go looking for that record before anyone knew it existed.

Early Pagedeck worked that way. Sofia would relay that three customers had written in about documents failing to save, and Theo would open that day's log file. A single day ran to four million lines, so filtering by the customer's account number and reading around each hit to guess at a cause took half a day.

The worse problem was the errors nobody wrote in about. An error that fired 4,200 times from one cause still appears in the log as 4,200 separate lines, each with its own timestamp and account number. So an error hitting 310 people sat in the file with the same weight as one that hit a single person once, and there was nothing to decide from.

Error tracking hands that job to a machine. Every time an exception fires, the service ships the details to a collection server, and the collection server merges everything that looks like the same cause into one entry. Instead of digging through raw records, a person reads a list that has already been grouped and counted, top down.

Its job is to turn scattered exceptions into units a team can fix, and it does four things to get there.

1. It collapses identical errors into one entry. For each incoming exception it combines the place it fired, the sequence of function calls that led there, and the error type into something like a fingerprint. A repeat of the same fingerprint just increments the count on the existing entry, so 4,200 lines fold into a single row reading "permission lookup failed during document save."
2. It attaches the material for deciding what to fix first. Every entry carries a total occurrence count and a separate count of affected users. One person hammering refresh can rack up 3,000 occurrences, and that entry does not outweigh one that hit 300 people once each. The second one goes first.
3. It stores the context needed to reproduce the problem — the stack trace (the list of function calls leading to the point of failure), the browser, the service version, the screen the user came from. A conversation that used to end at "it works on my machine" narrows to "only on Safari, and only on the sharing screen."
4. It tells you about errors you haven't seen before. When a new entry appears or a familiar one spikes, an alert goes out — an alert being the mechanism that contacts whoever is responsible once a condition you defined is crossed. A brand-new entry arriving at forty a minute ten minutes after a deploy means somebody has to decide right now whether to roll that deploy back.

Put those four together and the order of work inverts. Instead of waiting for a customer to report something, the team works down the list and fixes what nobody has complained about yet.

This is not the only way to find out about exceptions, and depending on how many you get, another approach can be the right one.

1. Keep logging and go digging when you need to. Logging writes down what happened during execution, one line at a time. It is the cheapest option and it already exists everywhere, but somebody has to go look, and it never counts repeats for you. For an internal tool throwing a few dozen exceptions a day, that is enough.
2. Watch the error rate as a metric. Metrics are observed values counted and stacked along a time axis. A metric pins down the minute a failure rate jumped from 0.2 percent to 14 percent, but it won't say which error jumped — so teams run it alongside error tracking, not instead of it.
3. Let application performance monitoring (APM) cover it. APM shows which screens and which requests are slow, organized around response time, and most products now list exceptions too, which means one fewer tool to run. The grouping rules and the ownership workflow tend to be finer-grained in a dedicated error tracker.
4. Build your own. Writing exceptions to a database and putting a screen on top of it is a day of work. The cost lands afterward: rules for what counts as the same error, a retention policy, throttling when something floods in, and a search screen that never stops needing attention.

"This issue fired 4,200 times today and hit 310 users. Let's fix it now, not next week."

Arguing for moving something up the queue based on the number of people affected rather than the raw count.

"A new issue showed up after yesterday's deploy, but it's still at three occurrences a day, so I'd rather watch it."

It's an error nobody has seen before, but not at a level that justifies rolling anything back.

"That one's a bot request, so I marked it ignored. It won't show in the list anymore."

Clearing out entries nobody intends to fix, which is what keeps the list readable.

At Pagedeck the half hour after a release is when this screen gets watched most closely. Theo filters down to entries created in the last hour, and when an unfamiliar one starts climbing at dozens a minute he passes the affected-user count straight to Mia. If that number stays in the double digits, they ship a hotfix for just that path. If it spreads into the hundreds, they roll the deploy back and raise an incident — an incident being an event that affected the service, which from there on brings customer communication and a written recovery record.

The most common assumption is that installing this means you'll know about every outage. Watching errors stack up in real time certainly feels that way. But the tool only catches cases where an exception was thrown. Requests that take eight seconds and then succeed, a screen that quietly renders a number a miscalculation produced, a server so dead it can't report anything — none of those ever reach the list. Watch only this screen and you'll learn about your quietest outages last. That's why it sits next to externally measured values like uptime.

The other one is treating an empty list as the goal. Leftover issues look like neglect, so the instinct is understandable. But some share of what users generate can't be fixed at all — someone hitting save on a subway train as the connection drops, for instance. Set zero as the target and the team starts marking things ignored instead of fixing them, and six months later the screen is clean while entries that deserved attention have been filtered out along with the noise. The number worth holding the team to isn't how many entries remain, but whether every new one gets triaged within a day.