Backfill

The flow that moves and reshapes data usually runs on a schedule with nobody watching. Batch processing is the common shape of it: every morning at three, the job collects yesterday's records and handles them in one pass. That flow only moves forward. The job running this morning looks at yesterday, not at last month.

So when a past range goes wrong, the flow cannot repair itself. There are two ways that happens. The first is failure: the source stopped responding for three days, or the job quietly died, leaving that stretch empty. The second is a change of rules: the values are there, but the calculation has since changed, so older records were built under the old definition and newer ones under the new one. Both leave a stretch of history that disagrees with everything around it.

Pagedeck ran into the first one. Mia opened the paid-conversion chart on a Monday morning and saw zeros from the previous Wednesday through Friday: the job that pulls payment records had been failing all three days. Theo fixed it that morning, but a fix only applies to later runs, so the empty days stayed empty. Naming a past range and putting it back through the pipeline is what the word backfill refers to.

A backfill pins down the range to refill, runs that range and nothing else, then checks the result against something independent. Six steps.

1. Fix the range. Write it as "May 1 through May 3" — a start and an end, as dates or timestamps. Starting from "wherever it looked weird" means nobody can say afterward how far the rerun reached, and a range ends up run twice or skipped.
2. Remove the cause first. If the range is empty because the source stopped responding, fix that before rerunning. Otherwise the job fails at the same point for the same reason.
3. Check how the pipeline writes. Find out whether rerunning a range deletes the existing rows and writes fresh ones, or adds on top of them. If it adds, May 1 revenue lands on the number already stored and shows up doubled. Producing the same result for a range no matter how many times it runs is called idempotency, and it is what makes a backfill safe.
4. Run one day first. Instead of pushing a full year through at once, run a single day and look at what comes out. If that day is wrong, the other 364 will be wrong the same way.
5. Split the rest and run it in order. Break the range into days or weeks and feed the pieces through in sequence. A workflow orchestrator — the tool that runs jobs in order and respects the dependencies between them — takes the list, starts each piece only after the previous one finishes, and stops on the first failure.
6. Reconcile the output. Payment counts should match the payment system's ledger; user counts should line up with the days on either side of the refilled range. Turning that comparison into a rule that runs on its own is data validation.

A backfill fits when the cause is gone and a rerun can be trusted to produce what a correct first run would have. It shows up in three places: filling a gap left by a pipeline that failed for a few days, recalculating the past year under a metric definition you just changed, and loading old records into their new shape after a migration moves a store to a different structure.

The limits take four shapes.

If the raw material is gone from the source, there is nothing to rerun. This is the wall teams hit most often. When an external service keeps only 90 days of logs, a range from six months ago cannot be rebuilt. The honest move is to mark it as having no value and leave it out of comparisons.

Recalculating the past under today's rules changes numbers people have already seen. Add a rule that excludes internal test accounts, backfill last quarter, and that quarter's signup count drops below what was on screen days earlier. The new number is more accurate, and it still needs an explanation.

A backfill competes for capacity with the jobs that run every day. Push a year through in one go and today's run queues behind it, so you fix yesterday while today goes missing. Large backfills get split into pieces and scheduled for quiet hours.

Pipelines built on stream processing handle records one at a time as they arrive, which makes carving out a single past range harder than in a batch job, so teams keep a separate path just for backfills. Either way, a backfill restores data quality after the fact — it is not what keeps numbers from going wrong in the first place.

"I backfilled last Wednesday through Friday, and the totals match the payment ledger."

The three empty days were reprocessed, and the result was checked against an independent source rather than assumed correct.

"Changing this metric definition means backfilling a year, but the source logs only go back 90 days."

The recalculation is wanted but the raw material caps it at roughly three months.

"Let's run the backfill overnight, in pieces. Pushing it all through midday will delay today's run."

A proposal to time and size the rerun so repairing the past doesn't crowd out the daily flow.

At Pagedeck the word comes up in two situations. The first is right after an incident. Theo fixes the cause, splits the empty range into single days, and leaves it running overnight; the next morning Mia checks whether the refilled values line up with the days around them. The backfill is the last step of the recovery, not the first.

The second is when a metric changes. If Mia proposes a new rule for what counts as a paid conversion, Theo's first question is how far back it applies. Last quarter is half a day of work; two years means checking source retention before anything can be promised.

The most common mistake is treating a backfill as a recovery button that one command takes care of. It does start as one command, so it looks that way. But checking how the pipeline writes, confirming the source still holds the data, splitting the range, and reconciling the output afterward belong to the same job. Skip them and you either publish a table where revenue is counted twice or find today's run half a day behind.

The other common belief is that a backfill restores the past to the way it was. Refilling something feels like undoing something. But the values it produces are not the old values — they are new ones built by today's code under today's definitions, and if any rule changed in between, the two disagree. Run it without knowing that, and nobody can explain why yesterday's screen and today's disagree about the same week. That is why teams record which ranges were backfilled, when, and under which definition.