Application Performance Monitoring

"It's been slow lately" used to be the hardest kind of report for an engineering team to act on. Open the server resource graphs and the CPU sits at 30 percent with half the memory free. The machine looks fine while the user waits eight seconds.

Resource graphs can't answer the question because they measure the wrong place. They describe how busy the hardware is, not how many seconds passed between a user pressing a button and the screen responding. The next tool people reached for was logging — writing out what happened, one line per event. Log a start time and an end time and you can recover how long a request took. The catch is volume: those lines pile up by the millions each day. Finding the slow screen means grouping millions of lines by request type and computing averages and distributions, which leaves the team building and maintaining that pipeline itself.

Application performance monitoring, usually shortened to APM, is that pipeline sold as a product. It sits inside the application code, times each request from arrival to response, groups those times by request type, and lists them slowest first. The shift is in where the measurement happens — on the request rather than on the machine.

The first question it answers is which request is slow. Response times get grouped by screen or route and sorted, so the fact that the document list renders in 0.3 seconds while document save takes 4.2 lands at the top of the list. There's now one place to look.

The second is how many people that slowness happens to. A request that averages a comfortable 1.2 seconds may be taking 9 seconds in its top 1 percent. A percentile is the position of a value once you line all the values up smallest to largest, and performance work usually watches the 95th or 99th. That figure is how long the slowest one or five users out of a hundred waited. Averages hide a handful of long requests behind a crowd of short ones, which is how complaints accumulate while the dashboard stays green.

The third is which step inside the request is slow. Open that 4.2-second save and it breaks into 3.1 seconds of database queries, 0.8 seconds of permission checks, and 0.3 seconds of everything else. When the database is the culprit, many products will name the specific query.

The fourth is when it started. Plot deploy times on the response-time graph and you can see the value climb from 0.4 to 1.9 seconds right after Tuesday's 4pm release, which narrows the suspects to whatever shipped in it. Without that overlay the same investigation starts from "sometime in the last two weeks," and two weeks of commits is a very different search than one release.

The same work can be split across other tools. What differs is what you give up.

1. Measuring in logs yourself — record the elapsed time per request and aggregate the logs in a search tool. It's cheap and you choose exactly what to record, but the ranking view and the percentile math become yours to build and keep running.
2. Metrics and dashboards — counters like requests per minute and response time stacked on a time axis. They pin down when things changed to the second, but which request and which step inside it aren't in the graph.
3. Distributed tracing — stitches together the path one request took across several services in order. When the system is split into many services this is the more precise view, and most APM products now include tracing, so the line between the two is blurry.
4. Error tracking — a different target entirely. It collects what broke; APM collects what's slow. Both are needed and neither substitutes for the other. A request that returns an error in 80 milliseconds never shows up in a slowest-first list at all.

With one service and a few hundred users, option 1 holds up for a long time. Once there are dozens of screens and someone starts asking which one is slow, that's the moment to add APM.

"APM says the 95th percentile on save jumped from 1.1 seconds yesterday to 4.8 today."

The what and the when are already pinned down, so the next step is looking at deploys in that window.

"The average on that screen is fine but the tail is bad. Let me pull just the slow requests in APM."

Isolating the minority of long requests that the average hides.

"APM doesn't show that step. We never instrumented it."

The step falls outside what the tool times automatically, so the code needs a measurement point added.

Pagedeck got eleven reports of slow document saves over two days. Theo opened APM and found the 95th percentile on save had gone from 1.1 to 4.8 seconds, with the bend in the graph lining up with last Thursday's release. Breaking the request apart showed that 3.4 of the added 3.7 seconds sat in the permission check.

Mia uses the same view for customer replies. She checks what share of users fall into the affected range, then writes to the customer with a cause and an expected fix time. Being able to write "3 percent of save requests" where the message used to say "some users" is what the tool changed for her.

Some people expect installing APM to make the service faster. The word performance is in the name, and response times really do improve in the month after adoption, so the assumption is easy to form. What improved, though, was the code the team fixed after the tool pointed at it. The tool is a thermometer, not a fever reducer. Install it, let nobody open the dashboard, and response times stay exactly where they were while the subscription bill arrives every month.

Expecting full automatic coverage is the other one that bites. APM times the widely shared points — the start and end of a web request, database queries, outbound calls to other services — with no work from you. Custom calculation steps and internal job queues, though, only appear to the extent that someone placed a measurement point there. That's why a 4.2-second request sometimes opens into a single undifferentiated block labeled "3.5 seconds, other." Adopt the tool without instrumenting those paths and the gap shows up exactly when an incident makes it expensive.