Background Job

For a long time a web service did one thing: take a request, do the work, send a response. Press a button and nothing appeared on screen until the server had finished everything it had been asked to do. A server is the program that receives requests, processes them, and returns responses. While the work took a fifth of a second, nobody minded the arrangement.

The trouble showed up as soon as the work got long. When Pagedeck first shipped an export that turned a 300-page document into a PDF, the export took 40 seconds. Browsers give up on a response after roughly 30. Users watched a spinner, got an error page, pressed the button again, and hit the same wall.

The reports that reached Mia all had the same shape: "I clicked export and nothing happens." The server was in fact working. There was simply no channel to say so. From the outside, working and broken looked identical.

The fix was not to make the work faster. It was to stop making anyone wait for it. Return an acknowledgement right away, and do the real processing afterwards. The piece of work that gets peeled off this way is the background job.

Instead of finishing the work where the request arrives, the server writes down what needs doing, returns a response immediately, and lets a separate program pick the note up and run it. Four steps.

1. Intake — the server records the request as a job: "export to PDF, document 4821, requested by Mia." Writing that record takes about 0.05 seconds.
2. Response — the server answers on the spot with something like "accepted, job 9f3c." The user's screen is free from here on.
3. Execution — a program called a worker pulls the record and does the actual work. A worker takes no user requests of its own; it only drains the pile of queued jobs.
4. Delivery — when the work finishes, the result is stored and announced. The screen asks for status every few seconds, or the server pushes a notification first.

It has the shape of ordering at a coffee shop, where you get a receipt and a pager and go sit down. The ordering line and the machine that makes the drink are separate, so the person behind you can order even if the drink in front takes a while.

Every job carries a state: queued, running, done, failed. States are what let the screen report progress and what let an operator re-run only the jobs that broke. A failed job is usually retried automatically two or three times, a few minutes apart.

Retries come with a cost. If the work actually succeeded and only the path back was cut, the system records a failure and runs it again. That is why jobs that run out of sight are built to be idempotent — running the same request several times leaves the same result as running it once. Three copies of a payment receipt landing in one inbox is what the absence of that property looks like.

Anything that cannot finish inside a response window, and anything whose result the user has no reason to see right now, moves to the back. At Pagedeck it falls into four groups.

1. Email and notifications — each message goes through an external delivery provider and takes two or three seconds. An announcement to 100 people runs past four minutes, and no screen can be held open that long.
2. File generation and conversion — PDF exports, thumbnails for uploaded images, and usage-history spreadsheet downloads all sit here.
3. Calls to outside systems — once a payment is approved, the record is handed to the accounting system. If that system slows down or drops for a minute, the Pagedeck screen should not stall with it, so the call is pulled out of the response.
4. Aggregation and cleanup — yesterday's usage is totalled by team, and temporary files older than 30 days are deleted. Chain jobs like these in order to produce analysis-ready data and you have a data pipeline.

One question decides what moves back: does the user need this result on the screen in front of them right now? Renaming a document has to show the new name immediately, so it stays inside the response. A 300-page PDF is fine to collect once it exists, so it goes to the back.

The same line separates judgment from execution. Rules such as "the free plan allows three documents" — the business logic of the product — are checked inside the response, because a request that will be rejected is better rejected before it is ever accepted. Only the heavy work that follows a passing check moves out of the request.

"Let's push this to the background. Doing it inside the response will time out."

The processing takes longer than a browser will wait, so the request and the work should be separated.

"Background jobs have been backed up for three hours. I think we need more workers."

Jobs are arriving faster than they are being drained, and the processing side needs more capacity.

"That email goes out as a background job, so it's normal if it doesn't arrive the second you click."

The delay is designed behavior, not a fault.

When Pagedeck rebuilt the export, Theo brought the 40-second measurement to the table and Mia did not ask how to make it 10. She asked what the user was supposed to do with those 40 seconds. The answer was to show "we're building it" the moment the button is pressed and send a notification when the file is ready.

A week later Sofia relayed a customer question: would a 500-page document work too? Back when the export ran inside the response, the answer had to be no. With the work moved out, it became a job that simply takes longer.

The most common reading is that moving work to the background makes it faster. The screen comes back the instant the button is pressed, so it can hardly look like anything else. But building the PDF still takes 40 seconds. What moved is the person waiting, not the processing time. Miss that distinction and the slow part never gets fixed — the problem surfaces on a Monday morning when requests pile up and the queue stretches to 30 minutes.

The other one is assuming that because a job runs out of sight, its failures are invisible in a harmless way. No error appears on screen, so it feels like nothing happened. The opposite is true. A failure with no response attached is a failure nobody sees, and without a failure log and an alert of its own, customers find the incident before the engineering team does. By then "I paid but no receipt came" has been accumulating in the support inbox for days.