Stream Processing

Most metrics are fine computed once a day. Knowing yesterday's signups this morning is enough, which is why batch processing, gathering what accumulated and handling it on a fixed schedule, was the default for a long time.

That approach jams the moment something has to be known within seconds. At Pagedeck there was a stretch where a plan took up to six hours to actually change after a payment was approved, because the job that reads payment records and updates account tiers ran once, at three in the morning. A customer who paid at two in the afternoon couldn't use paid features that day; the money had left and the screen hadn't changed, and it stayed that way overnight. Sofia handled the support messages that arrived in between, one at a time.

The first attempt is always shortening the schedule — every three hours, every hour, every five minutes. But as the interval shrinks, the time one run takes approaches the interval itself, and each run ends up waiting on the one before it.

Stream processing removes the schedule instead. One event arrives, it gets handled on the spot and passed onward. Under data pipeline, the flow that moves source data into a shape you can analyze, it belongs to the cadence group.

Five things have to be settled before it runs.

1. List only the values needed within seconds. Not everything moves over. At Pagedeck the list holds three items: payment approval, plan changes, and accepted teammate invitations.
2. Queue the incoming events. A message queue, which lines work up for asynchronous handling, sits there. It's what keeps events from disappearing when the consumer slows or stops for a moment, and without it, wiring things together directly means the data for however long the consumer was dead is simply gone.
3. Cut the aggregation range into time windows. A stream has no end, so there's no such value as "how many so far." Set a window — the last five minutes, the last hour — and count inside it. The window length is the period the number refers to.
4. Decide what happens to events that arrive late. A phone that was switched off uploads its records ten minutes afterward. When an event belonging to a window arrives after that window closed, you have to choose between discarding it, holding the window open a little longer, or revising the value after the fact. Leave that undecided and the same hour's number differs depending on when somebody looks.
5. Prevent the same event being handled twice. A queue guarantees that events don't vanish, and in exchange it may deliver one twice. Without pipeline idempotency, the property that rerunning the same range doesn't distort the result, one payment lands in revenue twice.

It fits values where being late costs something: payments and tier changes, inventory counts, incident detection, fraud detection. There's one test — if this number is thirty minutes late, who does the wrong thing because of it. With no answer, it stays on batch.

There are four limits.

1. Partial reprocessing is awkward. Recomputing only two to three in the afternoon yesterday is harder than in batch. Doing a backfill, reprocessing a past range to fill it in, from a stream usually means maintaining a second copy of the same computation as a batch job.
2. Building and changing it costs more. It's a program that runs permanently, so changing the code means deciding how the aggregation state it currently holds gets carried over. With batch, the new code simply runs from the next cycle.
3. Changing a definition doesn't move the past. Decide today to exclude test accounts and only values from today forward change, while last month's stay on the old definition.
4. Most metrics don't need it. Put a value that's fine once a day onto this and nothing changes except the compute bill and the operational load.

Where a workflow orchestrator, holding the order of jobs and running them in sequence, catches failures on the batch side, monitoring and alerting play that part for a stream. Processing lag gets measured, and when the lag grows, somebody gets called.

"Payment updates take six hours. Let's pull just this value onto a stream."

Proposing to move one value rather than everything.

"How are late events being handled? Yesterday's number changed today."

Asking for the rule covering events that arrive after a window closes.

"The queue can deliver the same message twice — can you confirm duplicate handling is blocked?"

Pointing out the risk of values inflating through double processing.

At Pagedeck, Theo picks the values to move and Mia writes down what happens when each one is late. That document is the reason the list stopped at three. Most of the numbers turned out to be ones where thirty minutes of delay changed nobody's decision.

Separating the live screen from the accurate value was settled then too. Today's numbers on the dashboard come from the stream, and the values used in month-end reconciliation come from what batch recomputed the following day.

The most common reading is that immediate handling is also more accurate, because fast and correct feel like they travel together. It's often the reverse. Because of events that arrive late, the value a stream produced diverges from what batch computes for the same period afterward. Today's revenue seen at three in the afternoon differing from yesterday's revenue on the next day's reconciliation sheet is common, and the later one is right. So the two get used for different purposes: the fast value for seeing what's happening now, and the slower value for reconciliation and reporting.

The other frequent belief is that this replaces batch, carried by the impression that whatever arrived later pushes out what came before. But the two aren't competing for the same seat. Most teams start on batch, peel off only the places that have to be known within seconds, and run two sets from then on. Write everything as a stream from the start and the cost of building and changing it multiplies, and keeping data quality becomes much harder too, because once a wrong value is found there's no convenient way to go back and recompute.