Data Engineer

While only one or two people use the data, nobody needs to own it. An analyst with a question queries the database the product runs on, downloads payment records from the payment provider's dashboard, downloads ad performance from the ad manager, and stitches it together in a spreadsheet. Half a day and there's an answer.

That arrangement breaks in three places at once. First, heavy queries against the production database slow down the product itself — the month-end rollup runs in the afternoon and support starts getting tickets about documents saving slowly. Second, manual stitching multiplies with headcount: five people export the same data, combine it five different ways, and walk into the meeting with five different numbers. Third, sources change quietly. The payment provider renames a field, nobody hears about it, and next month's totals are wrong in a way nothing announces.

The data engineer role exists so one person owns all three. The paths everyone was walking separately get merged into one that runs on a schedule, and somebody becomes accountable for whether it ran correctly today. The output of the role is a state of affairs: the tables analysts open are already filled in when they get to work.

More of the day goes to keeping things running than to building new things. It tends to go in this order.

1. Check what failed overnight. The first thing to look at in the morning is the result of last night's data pipeline runs — the flows that carry data from its sources into the analytics store. One stalled job out of ten means a dashboard is quietly showing yesterday's numbers to whoever opens it. Getting it rerun before the morning meeting comes ahead of everything else.
2. React to source changes. The backend adds a field to an event, or an external service changes its response format, and loading breaks at that point. Most of these arrive without warning, so you learn about them after they break.
3. Work new requests. Something like "I want ad spend broken out by account" turns into connecting a source and building a new table in the data warehouse, the store where analysis-ready data is organized and kept.
4. Cut cost and runtime. When a query takes 20 minutes or the storage bill doubles in a month, the cause gets tracked down. A common finding is a job that only needs to process yesterday reprocessing the entire history every night.
5. Write the descriptions. Record what each column of a new table means in the data catalog. Postpone it and in six months that table is one nobody can use.

Items 2 and 4 consume the most time while being the least visible. Building a new table gets noticed by the person who asked for it; reviving a broken flow every morning looks exactly like nothing having happened.

The role takes data from one side and hands it to the other, which puts it next to both.

1. Backend developers. This is where data is born. What user actions get recorded, under what names, and whether field changes get announced in advance all get settled here. Without that agreement, item 2 comes back every week.
2. Data analysts. They use the tables and they file the requests. The shape a table needs to take gets worked out by listening to the questions being asked of it.
3. Product managers. They settle metric definitions. Whether active users means people who logged in or people who created a document has to be decided before the table can be built.
4. Legal. They decide how far personally identifying fields travel into the analytics store. Copying national ID numbers that no analysis needs adds nothing but a thing to protect.

Design conversations belong before the table exists. Changing what a column means after it's full quietly breaks every report built on it.

"The overnight load failed at 3am, so today's dashboard only reflects through yesterday."

The numbers aren't wrong, they're late, so don't read them as current.

"To add that column we have to start at the source. It isn't in the event at all right now."

The problem isn't retrieving it from the store — it was never collected.

"This query re-reads the full history every run. Let's make it append one day at a time."

The same result is available for a fraction of the cost.

Pagedeck has no dedicated data engineer, so Theo carries the work alongside his server work. For Mia's weekly metrics, records from signups, payments and document creation have to land in one table, and at first Theo merged three files by hand every Monday morning. As the meeting cadence grew, that turned into three hours a week.

So he built a flow that moves and joins the three sources automatically every night, with an alert when it fails. The three hours on Monday disappeared, and in their place came the occasional early-morning alert — the day the payment provider changed its response format, for instance. The work didn't go away. It changed from merging files by hand into keeping a flow alive.

The most common assumption is that handling data means doing the analysis too. The word data is in the title and the person does stare at numbers all day, so it reads that way. But what the role is accountable for is not the interpretation of values, it's that values arrive on time in the right shape. Explaining why signups dropped from last week belongs to an analyst; making sure the table holding that number is filled in again this morning belongs to the data engineer. Hand both to one person and the analysis requests, always the more urgent of the two, take all the time — while a table whose freshness quietly broke sits unnoticed for a month.

The second is treating it as a build project that finishes. The word pipeline suggests plumbing: install it once and it lasts. In reality the sources keep moving. External services change response formats, new features emit new events, and a job that took 3 minutes takes 30 once enough history accumulates. Declare the build done and move the person elsewhere, and six months later half the flows are stopped — at which point the first task is working out which tables are still alive.