Data Warehouse

Company data does not start out in one place. Signups and document history sit in the store the product runs on, payment records sit with the payment processor, ad spend sits in the ad platform. Each system holds data in whatever shape its own job requires, and nobody agreed on how to match an account in one system to the same account in another.

So a single question takes days. When Grace asks which channel last month's paid conversions came in through, Mia pulls signup sources from the product side, pulls the conversion list from the payment side, and matches them by email address in a spreadsheet. It takes two days. Next month the same question takes two days again.

When it's urgent, someone runs the query straight against the store the product uses. Theo once counted six months of document creation in a single query, and for the few minutes it ran the service got noticeably slower for everyone. When the store that runs the product and the store you analyze are the same store, the product pays for every analysis.

A data warehouse handles both problems at once. It gathers the scattered pieces ahead of time, and it keeps that copy away from the product.

It solves three problems: data sitting in separate systems, analysis slowing the product down, and departments defining the same number differently.

1. It pulls scattered sources into one place. A data pipeline runs on a schedule, visits the payment, advertising and support systems, and loads what it finds into the warehouse. Reconciling account identifiers into one key happens on the same pass.
2. It leaves the product's own store alone. A database handles the reads and writes the product needs right now; the warehouse only aggregates over a copy. A query that counts six months of activity can be as heavy as it likes and the service stays fast.
3. It makes one answer for one number. Put the revenue calculation into a single table in the warehouse and the dashboard, the weekly report and the investor deck all read that table. The principle is called a single source of truth, and without it the same month's revenue shows up three different ways in one meeting.

What makes all three possible is deciding the structure before the data goes in. Payment records get four columns — date, account, plan, amount — and whatever shape a source hands over gets mapped into those four. The cost shows up later: when a source adds a column it never had, or changes its format, someone has to adjust the load before it can be accepted.

There are four ways to get by without building a warehouse, and the choice turns on how much data there is and what shape it comes in.

1. Spreadsheets — with two or three sources and tens of thousands of rows, this is the fastest option available. A five-person company that builds a warehouse first has usually moved too early. Past a few hundred thousand rows the file stops opening, and because a person does the updating, nobody can tell which copy anyone is looking at.
2. Querying the product store directly — asking the live store with SQL, the standard language for pulling values out of table-shaped data by attaching conditions to a query. Nothing gets moved, so the values are always current. The load problem above stays exactly where it was, and data that lives outside the product, like payments and advertising, isn't there to query in the first place.
3. A data lake — raw files kept in whatever format they arrived in. This fits when a lot of the data can't be given a shape in advance: server log files, external responses whose fields differ from call to call. The trade is that the work of interpreting what a value means moves to the moment you read it.
4. A data mart — a small store holding only what one department or subject needs. It isn't a replacement for a warehouse, since a mart is usually carved out of one, so treating the two as competing choices misreads the relationship.

Lakes and marts sit alongside a warehouse more often than they replace it. A common arrangement keeps every raw file in the lake, cleans up the portion worth analyzing and loads that into the warehouse, then carves the frequently read slices into marts.

Three conditions together mark the switching point. The same question comes back every month, answering it means digging through three or more sources, and the work takes more than a day. When only one of the three is true, it's usually still early.

"Is that number from the warehouse or pulled straight out of the payment system?"

The same metric can come out differently in two places, so it's worth knowing which one someone read.

"That's not in the warehouse yet. Wiring it into the pipeline takes about a week."

The source has the data, but nothing is carrying it over yet, so it can't be read today.

"The warehouse is as of last night, so anything from today won't show up."

Loads run once a day, which means today's data appears tomorrow morning.

Pagedeck built its warehouse right after the Series A. Before that, Mia spent three days at the start of every month inside a spreadsheet, and Sofia had to ask her whenever she needed payment status by account. Now signups, document usage, payments and ad spend land in one place once a day, and Sofia reads it from her own screen.

Speed wasn't the only thing that changed. Mia's table and Sofia's table used to disagree about how many paid accounts closed last month, and half a meeting would go to settling which one was right. Now both of them read the same table.

People often expect a warehouse to make their data accurate. Loading it into an organized structure sounds like the values get checked along the way. What it actually does is move and collect: a source that hands over a wrong value produces a wrong value sitting neatly in a column. Leaving that belief in place makes things worse than before. In the spreadsheet era a strange value stood out, but a number that came through the warehouse reads as vetted, so it goes straight into the report the CEO sees.

The other expectation that breaks down is that a warehouse gets used once it exists. Everything is in one place, so surely anyone can find it — except that at three hundred tables a newcomer has no way to tell which one answers their question. Unless someone also writes down what each table holds and who maintains it, people fall back on asking the data team, which is the process the warehouse was supposed to replace.