Data Catalog

Once data starts piling up, the count of tables becomes a smaller problem than their names. A company has exactly one table of signups for about as long as it has one engineer, and six months later there are four tables with similar names. The only way for an analyst to know which one to use was to ask whoever built it.

That person isn't always around. When the author moves to another team or leaves, the table becomes unusable — rows keep landing in it every night, but nobody can say what the columns count. So the analyst who joined last month builds one more table that computes roughly the same thing, and now there are five.

The worse case is when the meanings quietly diverge. A column called active accounts counts people who logged in on one table and people who created at least one document on another. Both carry the same label, so two numbers show up in the same meeting and there's no way to settle it in the room. A data catalog takes the explanation that used to live in one person's head and writes it down where everyone can read it.

Each entry describes one table, with the following attached to it. A list of names alone is not a catalog.

1. Table name and a one-line description. Say what one row represents. "One row is one payment attempt" is the part that matters, because the same payments table totals differently depending on whether a row is an attempt or a successful charge.
2. Column meanings. Spell out what each column holds — this is where you pin down whether active accounts means logged in or created a document. Recording the allowed values helps too, as in: the plan column only ever contains free, team, or enterprise.
3. An owner. One named person responsible for the table — a person, not a team, because a table owned by a team is a table owned by nobody. That's who you contact when a value looks wrong, and leaving the field blank means nothing gets fixed when it breaks.
4. Refresh schedule and last refresh time. Hourly or daily, and how current the values you're looking at are. This is what freshness is judged from, and it prevents someone from reading a once-a-day table at ten in the morning as today's number.
5. A personal-data flag. Mark the columns holding personally identifiable information (PII) — values that can single out one person. Flagged columns get handled differently: narrower access, separate retention, stripped out of exports.
6. Example usage. Attach one or two representative queries. A query that actually runs teaches the table faster than ten lines of prose.

The split between who fills it in and who reads it shapes what this document has to be. Everything a machine can find out — table names, column lists, last refresh times — gets scraped from the warehouse automatically by a data engineer. What people type by hand is the part no scraper can reach: what a column means, what one row represents, and when the table should not be used.

That handwritten part belongs to whoever built the table, and that's where catalogs usually fall apart. For the author it's writing down something they already know, and skipping it has no consequence that day. Which is why many teams block a new table from shipping until its column descriptions exist.

The readers outnumber the writers by a wide margin, and they arrive knowing nothing about the table. That asymmetry is why the entries are written as answers to questions rather than as technical notes: which table, which column, as of when. The readers are everyone who didn't build the table. It's where a data analyst decides which table to open for a new question, where a product manager checks the definition behind a dashboard number, and where someone a week into the job can start without asking anyone. When the person hearing a question for the second time can answer with a link instead of an explanation, the catalog is doing its job.

"Which table did that number come from? Who's listed as the owner in the catalog?"

The value looks off, so find the responsible person and check with them.

"Active accounts is defined differently in those two tables. Let's go with what the catalog says."

Treat the written definition as the single reference rather than re-arguing it.

"We set it up so a new table can't ship until the column descriptions are filled in."

Descriptions deferred to later never get written.

Pagedeck built its first catalog after a weekly metrics review where active accounts appeared as two different numbers. Mia's deck counted people who had logged in; the marketing deck counted people who had created a document. Neither was wrong, but both carried the same label, and half an hour of the meeting went into working out which number to believe.

The following week Theo scraped the table and column names out of the warehouse to make a skeleton, then wrote one owner next to each table. Filling in the meanings stayed with the owners. Two months later the visible change wasn't accuracy — it was volume of questions. "Can I use this table?" dropped by more than half.

The common expectation is that installing a catalog makes the data correct. It gets introduced as a data quality tool, so the assumption is reasonable. But the catalog is a document of explanations and never touches the values themselves. A table that aggregates incorrectly goes on aggregating incorrectly after it's listed. Checking values against rules is the job of data validation, and the catalog's contribution is telling you who should set those checks up. A catalog and a set of checks solve adjacent problems — one tells you which table to use, the other tells you whether today's rows in it can be trusted. List everything and check nothing, and wrong numbers circulate with well-written descriptions attached.

The second is assuming it maintains itself. The automatic scraping makes that easy to believe. In practice columns get added and meanings shift faster than anyone updates the prose. Leave it alone for six months and some entries no longer match reality — and a reader who gets burned once by a stale description goes back to asking people directly and never returns. The document survives while its readership doesn't. That's why naming an owner for every table is settled under data governance before the catalog is started.