Anomaly Detection

A dashboard gathers the key numbers onto one screen. The open question is who looks at that screen and when. Twenty metrics can be scanned every morning, but once the company grows into eight dashboards with ten charts each, nobody checks them all daily. Usually last week's numbers get opened for the first time while someone preps for the Monday meeting.

So this happens. Payment failures climb from 1 percent to 4 percent after a Tuesday-night fix to the billing module, and three days pass with nobody aware of it. Sales forwards a customer complaint on Friday afternoon, someone finally opens the chart, and the line has been visibly elevated since Tuesday. The data was there the whole time. Only the reader was missing.

Anomaly detection reverses the order. Instead of a person going to the screen to find a problem, the machine comes to the person the moment a value falls outside its usual range. It leans on a simple asymmetry: watching every chart every day is a poor fit for humans and no burden at all for a machine.

Setting it up takes five steps, and the first one — narrowing what gets watched — decides whether the rest works.

1. Choose the metrics to watch. Not every chart on the screen, only the ones where a wrong value means somebody has to act that same day. Payment failure rate and signup completion rate qualify. Month-to-date revenue doesn't, because checking it a day later leads to the same response.
2. Define what counts as normal. Two approaches. A fixed threshold draws one line — payment failures above 2 percent. A statistical baseline collects the last 30 days of values for the same weekday and hour and measures how far today sits from that average. Metrics that swing widely within a day need the second.
3. Set the sensitivity. Tighten the band and small wobbles trigger it; widen it and real incidents slip through. Starting wide, at one or two alerts a day, and tightening each time something is missed is the safer direction.
4. Decide where an alert goes. Alerting is the machinery that contacts an owner once a condition is met, and anomaly detection is what hands that machinery its conditions. Without a named recipient and a route per metric, alerts pile up in a channel nobody reads.
5. Work the false positives down. Once a month, collect the alerts that turned out to be nothing. If a cause repeats, adjust the baseline or drop that metric from the list. Skip this cleanup and the alerts become background noise within weeks.

The procedure only stands if what's underneath it is sound. A shaky metric definition means the machine is measuring a moving target, and poor freshness — how recently the data was updated — means being told now about an incident that ended three hours ago.

Metrics that suit this share three traits: values arrive often, the usual shape repeats, and a deviation implies a clear action. Payment success rate, recorded hundreds of times a minute, has all three. Contract counts that land once a week have almost no comparable history, so the machine never gets to define normal. Four past values cannot say whether three this week is unusual or ordinary, and a baseline built on them flips between alarm and silence for reasons that have nothing to do with the business.

Complicated repeating patterns are also hard. A work tool loses half its usage on weekends, and a machine that doesn't know that rule reports an incident every Saturday. Comparing by weekday and hour solves it, but cycles that come a few times a year — holidays, year-end — stay broken because there isn't enough past to compare against. New products have the same issue. While users grow every week, today being higher than last week is normal, and there's no baseline to call usual yet.

The biggest limit is how narrow the answer is. This method answers exactly one question — is the current value unlike its usual self — and says nothing about why, or whether it matters. A spike in payment failures could be a card processor outage, last night's deploy, or one large customer switching payment methods. Getting from symptom to cause is root cause analysis, and a person does that separately.

"We have detection on it, and it fired at 11pm last night on a signup drop."

The machine caught a value outside the normal range and notified the owner.

"That one fired every weekend, so we pulled it out of detection for now."

The baseline couldn't account for weekday differences, so repeated false positives got it removed from the list.

"Detection is wired up, but nobody's assigned to receive the alert."

The condition exists; the person who would act on it doesn't.

Pagedeck runs anomaly detection on exactly three metrics: payment failure rate, signup completion rate, and response time. Last month at 2am on a Wednesday, payment failures hit four times the 30-day average for that hour and the alert fired. Theo, who was on call, confirmed it within 15 minutes and rolled back the previous evening's deploy. Around 40 customers were affected, all of whom had their retry succeed once the old build was back.

Mia was asked to add more metrics after that incident and declined. She set up a monthly review of everything that fired instead. The last review showed that half the response-time alerts clustered in the early-morning window when batch jobs run; excluding that window took the rate from nine alerts a week down to two.

The most common belief is that turning detection on reduces incidents. Being watched feels like being protected. But this method reports a change that has already happened; it doesn't stop one from happening. What shrinks is not the number of incidents but the time until one is noticed, and going from three days to 15 minutes is a large gain that never reaches zero. Mistake it for prevention and the things that actually prevent — pre-deploy checks, staged rollouts — get postponed.

Believing that more alerts mean tighter coverage is the other one. Detection on thirty metrics looks safer than detection on twenty. In practice nobody reads a channel that fires fifteen times a day for more than two weeks. Once it goes unread, the real incident mixed in there passes by too, which leaves you where you'd be with no detection at all. The performance of this machinery isn't how many rules are armed but what fraction of alerts actually move someone.