Load Test

Whether a feature works and whether it works with a crowd on it are different questions. One person clicking once answers the first. Nothing about that answers the second. A document list that loads in 0.2 seconds alone can take 8 seconds when 2,000 people call it at the same moment, and those 8 seconds happen while every feature is functioning correctly. From the user's side there is no difference between that and being broken.

There is no way to learn that number without the crowd, because the cause only appears when requests arrive together and start contending for the same resources.

So for a long time, real users did the checking. Pagedeck learned it that way. On the first morning of a large customer's company-wide rollout, 300 people signed in at nine o'clock and the document list refused to open. Theo found it wasn't a bug: connections to the database had hit their configured limit and everything else was queued behind them. A database is the system that stores data and serves it back, and the number of connections that can be open against one at once is normally capped.

A load test moves that discovery from the customer to the team. You generate the traffic you expect to see, push it in, and watch which part gives way first from your own desk.

There are six steps, and skipping the first two leaves you with numbers you can't interpret.

1. Set the target. Write it as a number: "the list screen responds within 1 second at 2,000 concurrent users." If a service level objective (SLO) already exists, reuse its value. An SLO is the quality level a team has committed to holding. Run without a target and there's no basis for calling a result good or bad.
2. Choose the scenario. Rather than hammering every screen, mirror real usage. For Pagedeck that's roughly 70 percent opening the document list, 20 percent saving a document, 10 percent searching. Get the mix wrong and you go hunting for a bottleneck in the wrong place.
3. Take a baseline. Run about 50 users first and record what response times look like when nothing is strained.
4. Ramp up in stages. Don't start at 2,000 — go 200, 500, 1,000, 2,000, holding each level for five to ten minutes. That's how you find the user count where things bend. Dump it all in at once and the only fact you get back is "it died."
5. Read the results. Averages are nearly useless here: if nine people in ten are served in 0.2 seconds and one waits 20, the average still reads 2 seconds. Look at the value with the slowest 5 percent trimmed off, alongside the share of requests that errored and what the servers' processing load and memory were doing at the time.
6. Fix whatever bent first, then run it again. A bottleneck is the slowest stretch, the one that sets the pace for everything, so until it widens, work anywhere else leaves the overall number unchanged. If connections ran out, raise the limit; if the same query kept repeating, add caching; if there weren't enough servers, add machines behind the load balancer.

Fix it and the bottleneck moves somewhere else. That's the expected outcome. The cycle repeats until the number clears the target.

It fits when you know the crowd is coming. A company-wide rollout at a big customer, the first day of an ad campaign, a feature launch, month-end when billing piles up — anything with a date on it gets a run beforehand.

The limits are just as clear.

1. Generated requests are not people. A tool produces a fixed sequence repeated over and over, with none of the pausing, going back, or leaving tabs open that real users do. Passing therefore doesn't certify you against real traffic.
2. A different environment gives different numbers. When the test environment runs at half the production spec, it's tempting to double the results and call it done, and it doesn't work out that cleanly. Measuring on production-grade hardware is the accurate option, and that means arranging a run in the quiet hours.
3. You drag outside services in with you. Anything that calls someone else's server — a payment processor, an SMS gateway — will take real load from your test and can fail on their side. Swap those stretches for stand-ins before the run.

"Let's do a run at 2,000 before next week's rollout."

The date the crowd arrives is known, so check capacity before it.

"It's fine to 1,000 but bends at 1,500."

The limit has been located, and the next question is what gives way in that range.

"Let's read the trimmed value, not the average."

Slow requests are getting buried in the mean, so measure from the perspective of the people who actually waited.

Since that incident, Theo runs one every quarter at Pagedeck. At three in the morning he spins up an environment matching production, ramps to 2,000, and writes up where it bent along with the per-stage timings that showed up in APM. APM stands for application performance monitoring, the kind of tool that traces which part of a request consumed the time.

Passing once gets read as being in the clear. It feels like a certification, but it's a measurement of that code against that volume of data. A service that held 2,000 users with 100,000 documents in it can bend at the same user count once there are 3 million, because the query behind the list now has thirty times as much to scan. Measure once and shelve it and the same failure returns in the same place six months later.

The other common belief is that a bad result just means adding servers. Adding machines is the easiest lever, and with autoscaling turned on they even appear by themselves, so the reading is an easy one to reach. But if the bottleneck is one database, going from 4 front-end servers to 12 leaves response times where they were and adds connections that make things slower. The value of a load test isn't learning how many users you survive; it's learning what gives way first. Add machines without knowing that and the only number that moves is the bill.