Workflow Orchestrator

Running several jobs at set times was handled with a timetable for a long time. Write down that this runs at 3 a.m. and that runs at 3:20, and the machine executes at those times. A timetable is easy for people to read too, and with two or three jobs it is enough to run a whole reporting stack.

The day it broke at Pagedeck went like this. Pulling source data was scheduled at 3:00, cleaning it at 3:20, aggregating metrics at 3:40 — the extract, transform, load order known as ETL. One night the pull took 40 minutes instead of 20.

The cleaning job ran at 3:20 as scheduled. Not knowing the pull had not finished, it cleaned half-loaded data, and the aggregation ran on that result at 3:40. The dashboard Mia opened in the morning showed the previous day's revenue at half its real value.

Nothing having failed is the character of this incident. All three jobs completed their own work normally and no error was raised. The only thing that broke was an assumption about order, which no monitoring was checking. Express order as clock times and the same thing repeats every night a job runs longer than expected.

A tool that manages order by making the success of the previous job the condition, rather than a clock time, is what took hold.

Five things.

1. It writes order as a relationship. Instead of saying start at 3:20, you say start when the pull succeeds. With ten jobs where three of them have to wait on each other, a timetable has no way to express that; arrows do, and the graph reads the same way a year later.
2. It handles failure. How many times to retry, at what interval, and whether a final failure stops downstream jobs or lets them skip, get set per job. Jobs that fetch from an external source often succeed on the second attempt, so that setting alone removes a lot of manual work from somebody's morning.
3. It reruns parts. Failing at the fourth of five jobs and restarting from the top spends the first three jobs' time again. Being able to rerun from the failure point, and for yesterday's date only, is the practical value here. Whether the rerunnable unit was designed as a date decides whether that value is available.
4. It makes things visible. How long each job took and where it stopped stays on a screen. That is where you notice a job that has started taking longer than it did last week, before it becomes an incident.
5. It notifies. Not only on failure but also when something has not finished by a given time. The second matters more: failure surfaces as an error, while running late is invisible until somebody sees an empty dashboard in the morning — which is exactly what a lateness alert catches.

The shape of use is consistent. Each job is written as code and the dependencies between them are drawn as arrows into a graph. What a job actually does is a query or a script; this tool executes it and checks the result. Recalculating the past 30 days means the same graph with different dates rather than a separate script. Because the graph lives as code, what changed and when accumulates as history.

Four options, chosen by job count and depth of dependency.

1. A time-based scheduler. The simplest way to run a command at a set time, and the better choice when there are three or fewer jobs with nothing waiting on anything. The incident above is what happens when dependencies appear and this stays in place. There is nothing to learn beyond a name and a time.
2. A tool that manages transformations inside the warehouse. Under ELT — loading raw data first and transforming it inside the warehouse — such a tool works out the order between transformations itself. Its scope stops at transformation, so fetching from sources and alerting remain outside.
3. A managed service. The same capability rented rather than installed and operated. It fits a team with nobody to look after servers, at the cost of pricing that scales with job count and of being harder to leave.
4. Doing it by hand. Early on this is a real option. For two aggregations run once a week, the cost of adopting a tool exceeds the cost of running them yourself, and it will keep doing so for a while.

Three questions decide it: whether job count passes ten, whether anything waits on anything else, and whether a person needs to know in the morning that something failed. The third one is usually the trigger, because it is the one that costs somebody a morning.

"The pull ran late and the aggregation went anyway. We need a dependency."

Switch the condition from a clock time to the previous job's success.

"Rerun yesterday only, please. The full run takes two hours."

A request that shows why partial reruns matter.

"We get failure alerts but no lateness alerts."

The mechanism for catching a late run is missing.

At Pagedeck, Theo maintains the graph and Mia uses the results. When a new metric gets added, the first thing Theo asks is by when the value has to be ready. If it feeds a 10 a.m. meeting, the alert is set backward from that time; if it only has to land sometime that day, it goes on the list with no alert at all.

A growing graph becomes something to manage, since nothing removes a job on its own. Aggregations nobody uses keep running, so every quarter the outputs nobody opens get traced and those jobs come out of the graph.

Treating it as a tool that processes data is common. It is the first name mentioned in any pipeline discussion, so it reads that way. But all it does is manage order and confirm execution — it only sees whether a job reported success. Moving and cleaning the data is done by the code inside each job, and heavy computation usually runs in the data warehouse. Miss that distinction and you look for the cause of a slow aggregation in the tool, while the query that needs fixing stays exactly as it is.

The second is expecting a pipeline to become stable because the tool is in place. It ships failure handling, so that expectation is natural. But retry counts, lateness alerts, and rerun units all require someone to configure them. Left on defaults, the timetable has merely been redrawn as a graph, and the incident recurs in the same place. Half the value of adopting the tool comes from deciding those settings job by job.