Caching

A server used to redo the same work on every request. Opening a team's document list means pulling 200 documents out of the database, checking who is allowed to see each one, sorting them by last edit, and shaping the whole thing into something a screen can render. If a colleague opens the same list one second later, nothing has changed in that second, and the server does every one of those steps over again.

With one person using the product, the waste is invisible. It shows up when many people open the same screen at once. At nine on Monday morning, the whole company signs in and opens the document list, the server runs the same computation hundreds of times a second, and a response that normally takes 0.2 seconds stretches to three.

The usual first reaction is to add servers. More servers do push the slowdown further out. But each new server repeats the same computation, so the bill doubles while the proportion of wasted work stays exactly where it was. Caching comes at the problem from the other side. Instead of doing the computation faster or in more places, it holds on to the result of a computation already done and hands that back, skipping the work entirely.

Pick what to store, decide where to keep it, and decide when to throw it away — in that order. Caching without those three decisions is a machine for showing wrong screens quickly.

1. Pick what to store. Good candidates are expensive to produce, requested often, and changed rarely. A document list is requested thousands of times a day and edited a handful of times, so it fits. Who is currently viewing a document changes second by second, so it doesn't.
2. Decide where to keep it. A copy in the user's browser only speeds up that person's second visit. A copy in server memory benefits everyone. Keeping copies at regional points close to users is what a content delivery network (CDN) does — a user in Seoul never has to make the round trip to a server in the United States, which visibly cuts time to first byte (TTFB), the gap between sending a request and the first byte coming back.
3. Decide how long to hold it. This is normally written as a lifetime in seconds. A logo image that never changes can be given a year. A document list that might be edited gets 30 seconds or five minutes. Short lifetimes shrink the benefit; long ones raise the odds of showing something stale.
4. Decide how the stored copy gets thrown out when the original changes. You can wait for the lifetime to run out, or delete the copy the moment the underlying data changes. If the code that edits a document also clears that team's list cache, edits show up immediately. Miss one of those places and that screen keeps serving last week's content for days.
5. Check whether it is actually working. Look at the share of requests answered from stored copies. At 90 percent, nine out of ten computations the server used to run are gone. At 20 percent, either the lifetime is too short or the wrong thing was chosen to store.

Caching fits where reads vastly outnumber writes and where the same request can safely produce the same answer. A public help article gets read 50,000 times a day and edited two or three times a month, so nearly every request can be served from a stored copy.

It fits badly on screens whose content differs per person — your notification list, your billing history. You can still store those, but you have to store them per user, so the storage grows with the number of users while each user reuses their own copy barely at all.

Three limits are worth knowing. The first is staleness: when the original changes and the stored copy survives, users see deleted documents still sitting in a list. The second is that finding the copy to delete gets hard. Once copies live in three or four places, tracing which one has to be cleared for a change to appear takes longer than building the feature did. The third shows up when many copies expire at once. Give every copy the same lifetime and, at that instant, every request stampedes to the original at the same moment, shaking the server harder than it would have been shaken with no caching at all. Teams avoid this by scattering lifetimes by a few random seconds, or by rebuilding a copy just before it expires.

"That's cached, so even after you fix it people will see the old screen for another five minutes."

The screen isn't wrong; the team is waiting for a lifetime to run out.

"Let's put a cache in front of the list endpoint. I think we can cut the response time in half."

The same request keeps repeating, so let's skip the computation.

"We forgot the cache invalidation. Deleting a document has to clear the list too."

Naming an incident caused by an original changing with no code to remove the stored copy.

At Pagedeck, Theo put a five-minute cache on the document list response and the average response time dropped from 1.4 seconds to 0.3. Two days later a support ticket arrived: a customer had deleted a document and it was still in the list. The deletion code removed the document itself but never touched that team's cached list.

Theo fixed it by clearing the team's list cache from all three places that change it — delete, edit, and create. The hard part of caching is not storing things. It's throwing them away.

The most common mistake is treating caching as a fix for slow code. Response times fall the moment you turn it on, so it looks like one. But any request that can't be answered from a stored copy still runs at the original speed. Put a cache in front of a three-second query and the average looks like 0.4 seconds, while the first user of the day and anyone arriving just after a lifetime expires still waits the full three. Leave the slow query alone and those three seconds hide behind the average until a heavy traffic day brings them all out at once.

The other mistake is assuming that storing more makes things faster. Every additional stored copy is another copy someone has to remember to clear, and fewer people can say which screen is reading which copy. Eventually the team reaches a state where the data looks wrong and nobody knows where to delete. Choosing a narrow set of things to store, and writing down the rule for clearing each one, ends up faster.