Batch Processing

For a long time, using data meant someone doing it by hand. Every morning a person downloaded yesterday's transactions from the payment system, pulled a separate list of signups, and pasted the two together in a spreadsheet to build a revenue table. It took half an hour, so nobody treated it as a problem.

The problem was that the half hour came back every single day. When the person who did it took a week off, the numbers for that week were simply missing. When they pasted the columns in the wrong order, duplicate rows slipped in and revenue came out higher than it really was. Anyone who asked "is this number right?" in a meeting sent someone back to the files to rebuild the whole thing from scratch.

What stood out was that the content of those thirty minutes never changed. Same sources, same joins, same filters. Only the date moved. So the procedure could be written down once and handed to a machine that runs it at a fixed time. Batch processing is that idea with a name on it.

Pick a cadence, define the window of data each run covers, chain the work that processes that window, and attach a trigger. Five steps in practice.

1. Pick the cadence. Once a day or once an hour — the deciding factor is not how often the source data changes but how stale the reader can tolerate the number being. A revenue table read in the morning standup only needs one run a day.
2. Define the window. A job that fires at 3 a.m. covers yesterday from midnight to midnight. Pinning the window to exact timestamps is what keeps a run from counting yesterday twice or dropping the records that landed just before midnight.
3. Pull that window out of the source systems and land it in storage. Sources are where data is born — the payment system, the product database. Storage is a separate analytical data warehouse. The reason for moving it rather than querying in place is that heavy aggregate queries slow down the screens real users are looking at.
4. Chain the jobs that run inside storage. Linking payments to users has to finish before the revenue table can be built, so that ordering goes to a workflow orchestrator, which holds the dependency map and stops downstream work when an upstream job fails.
5. Attach the trigger. A fixed clock time like 3 a.m. needs nothing more than a cron job. If the run has to wait for a file to arrive, the arrival itself becomes the signal.

One more property belongs in the design from the start: idempotency, meaning a window can be reprocessed without the result drifting. Batches fail, and a failed batch gets rerun. If rerunning May 1 doubles that day's revenue, the repair becomes the incident. Backfilling a past range is only safe on the same foundation.

It fits anywhere a few hours of lag inconveniences nobody: yesterday's revenue, weekly active users, month-end billing, a list of accounts at risk of leaving. Most reporting metrics live here. The number needs to be on the screen by the 10 a.m. meeting, and nobody asks whether it was computed at three or at six.

The simplicity is worth as much as the cost. A closed window is easy to check against a known total, and when it comes out wrong you rerun that window and nothing else.

The limits show up in three shapes.

1. The cadence is a ceiling on freshness. In a daily pipeline, the oldest number on the dashboard is 24 hours old. Anything that has to move within seconds of a payment clearing cannot be served this way.
2. Failures arrive whole. When a per-record flow drops one record, one record is missing. When a batch dies midway, the entire day is either empty or half filled. Alerting and a documented rerun procedure are not optional extras here.
3. The work piles into one moment. Storage runs hot from 3 to 4 a.m. and idles the rest of the day. When every team schedules against the same hour, the jobs queue behind each other and finish after breakfast.

Which is why most teams start with batch and carve out stream processing only for the specific places that need an answer in seconds, leaving everything else alone.

"That figure runs on a batch, so this morning's orders aren't in it yet."

What's on the screen is the total through yesterday, not through right now.

"The overnight batch failed and the dashboard is stuck on yesterday. We're rerunning it."

A scheduled job died partway and the same window is being reprocessed.

"Does that really need to be real time? A once-a-day batch seems like plenty."

An argument for relaxing the requirement to cut build cost and operational load.

At Pagedeck, most of what Mia looks at before the morning meeting comes out of a 3 a.m. batch. One day the signup count for the previous day showed zero. The payment system had been down for maintenance, the extraction step failed, and everything downstream stopped behind it. Theo reran the same window once maintenance ended and the numbers filled in before noon.

Two things changed after that. Failures now post an alert, and every dashboard carries the timestamp of its last successful run at the top. A blank number is less dangerous than a stale one mistaken for fresh.

The most common misreading is that batch is the outdated option and real time is the modern one. Technical writing tends to lead with streaming, so the impression forms on its own. In practice the choice comes from the requirement, not from how current the technology is. Month-end billing cannot be real time because the month has to close before the math is possible. Teams carrying this assumption spend several times the budget streaming something a daily run would have covered, and then find that recovering from an outage — rerun yesterday, done — is no longer simple.

The other misreading is that batch means overnight. Nighttime examples are everywhere, but the cadence can be daily or every five minutes. Accumulating records and processing them together is the dividing line; the hour on the clock is not. Miss that and someone describes a five-minute batch as real time, and the person listening reports a bug when the screen shows a number from five minutes ago.