Freshness

Wrong numbers get caught quickly. When revenue drops to zero or signups go negative, somebody says something. Late numbers get caught by nobody. A plausible value sits on the screen, and whether it's today's or yesterday's is written nowhere.

It happened at Pagedeck. In the Monday morning meeting Mia read out weekly active users and said the count was up from last week, when in fact the job filling that table had failed on Friday night and never run again. Two full weekend days were missing from the figure on the screen, and none of the six people in the room had any way to know.

Freshness is the name given to making this kind of failure visible. Instead of asking whether the value is right, it asks when the value is from. Change the question and the screen tells you how far back the table's coverage ends. With that number, a late table can be called late; without it, the table looks fine right up until it's wrong.

Subtract the timestamp of the most recent data in the table from the current time; that gap is the freshness. The gap is called lag and is usually counted in minutes or hours.

If it's 2:20pm and the latest row in the session log table is stamped 1pm, the lag is one hour twenty minutes. For a table meant to refresh on the hour, that's normal. If the latest row in the same table is from 11pm last night, the lag is fifteen hours, which means the processing stopped somewhere.

Decide first which timestamp the measurement is based on. There are two candidates.

1. The time the event actually happened — the moment the user opened the document. It's honest, because it measures the lag as the customer's side produced it, but an app that reports its records late shows up in the lag too.
2. The time the value landed in the table. A data pipeline is the flow that moves scattered source data into an analyzable shape and cleans it up; this is the moment that flow loaded the value into the analytics table. Measured this way, you see immediately whether the pipeline is running, but old data loaded by a healthy pipeline still reads as fresh.

Many teams track both and read them together, since the difference between the two is itself information. An event-time lag of six hours with a load-time lag of five minutes means the pipeline is fine and the source is feeding it late.

The unit being measured is one table, not the warehouse. The payments table refreshes every 10 minutes, the session log hourly, customer account details once a day, so no single standard covers all of them. Record an expected interval per table and count the time past that interval separately: a table due hourly that has sat unchanged for three hours is two hours over.

Shorter lag isn't automatically better. The standard is the cadence of the decisions made from that table. A table used to build a quarterly budget can be a day behind without changing the decision, while a session table watched during an incident that's 20 minutes behind has you responding to a situation that already passed.

So each table gets a written tolerance. Pagedeck sorts them into three buckets.

1. 15 minutes — payments and incident response tables. Going over pages the owner right away.
2. 3 hours — general metrics on dashboards. One late morning rarely changes that day's decisions.
3. One day — tables like account details or plan history that don't change often anyway.

Writing the number down isn't enough on its own; decide what happens when it's exceeded. Either the owner gets an alert, or the screen shows the last update time and lets the reader judge. The second is cheap and effective. A line in the dashboard corner reading "as of 3 hours ago" ends the habit of reading a late table without knowing it's late.

Collect a month of measurements and a stale-once table turns out to need a different fix than a slightly-late-every-day table. For the first, trace back through the data lineage to find what happened that day — lineage being the recorded path showing which source a value came from and what processing it passed through on its way to the screen. The second says throughput already can't keep up with the interval, so the processing or the hardware needs work. Some teams manage it as a rate: 5 of this month's 720 refreshes running past 15 minutes gives 99.3 percent compliance.

"As of when is this dashboard? If those are morning numbers they don't match what you just said."

Without knowing how far the values on screen extend, the discussion can't continue.

"The number isn't wrong, it's late. A job failed overnight."

The values are accurate but frozen at yesterday's state.

"Let's put the payments table on the 15-minute standard and leave the rest at 3 hours."

A proposal to set different tolerances per table.

The phrase comes up most often in Pagedeck's Monday morning meeting. When Mia puts up the weekly metrics, Theo checks the update time in the corner first. For a while nobody did, so the team held two meetings on Friday evening numbers, and after that every dashboard got a last-updated stamp at the top.

Now a late table pages Theo before the meeting, and he opens with "payments hasn't landed today yet" the moment he sits down. Everyone agrees on when the numbers are from before reading them.

The most common misreading is that fresh means accurate. An update stamp from five minutes ago makes the table feel trustworthy, but freshness guarantees exactly one thing: the value isn't late. A payment amount recorded as negative in a row that arrived five minutes ago still has a lag of five minutes. That side is caught by data validation, which checks values against rules agreed in advance. On a table with only a freshness check, a strange value arrives on time and gets used.

The other frequent one is thinking more frequent refreshes everywhere solves it. If lag is the problem, remove the lag — it sounds reasonable. In practice, taking a refresh interval from daily to every five minutes multiplies processing cost many times over, and most tables built that way get opened once a day, with fewer than that changing any decision. The thing to decide isn't how fast, but how many hours late the decision made from this table can tolerate.