Data Quality

For a while, companies treated collecting data as the hard part. Build a data pipeline that pulls values out of scattered sources, reshapes them and moves them to one place, add an analytical data store to hold what arrives, and the rest looked like a matter of querying it.

What actually happened looked more like this. A Monday metrics meeting opens with a dashboard showing 1,240 signups for the previous week, while the number sales counted separately comes to 1,190. Twenty minutes go into arguing about which one is right and nobody wins. Three days later somebody discovers that one sync job failed early Saturday morning and 50 signups from that day never landed.

The annoying part is that nothing looked empty. The screen displayed 1,240 with no warning attached, and anyone reading it had no reason to treat it as anything other than the whole number. Missing rows get undercounted. A user who arrives as two rows gets counted twice. A blank amount field counts as zero. The query succeeds every time and produces a result every time. The only thing it never produces is a signal that the result is wrong.

The 50 missing signups were not the expense. The expense was the silence around them. That week's notes recorded that signups had fallen, and a suggestion to trim ad spend followed from it. The cause surfaced three days later, but nobody went back and redid the judgments made in between. A wrong number can be corrected; the decisions made while looking at it stay where they are.

The pattern repeats because nothing about a broken number announces itself. A pipeline that crashes outright gets noticed within the hour, since the table simply stops growing and somebody complains. A pipeline that keeps running while one of its five sources quietly returns nothing can go a month unnoticed, because the table keeps growing and the total keeps looking plausible. The failures that cost the most are the ones that leave the output looking normal.

Once a team has lived through this, it starts every meeting by doubting the numbers. When doubt becomes the habit, the dashboard has stopped earning its place. A figure you have to verify separately each time you see it is not far from a figure you don't have.

Data quality is the name for pulling that verification out of individual heads and turning it into something that watches continuously. The weight of the term sits less on catching a bad value after the fact and more on writing down in advance what counts as bad, then arranging for someone to hear about it when reality drifts from that definition.

The terms under data quality fall into three groups: measuring whether the values themselves are usable (checking), following where a value came from and where it currently lives (tracing), and setting who may use what and how far (stewardship).

Checking measures the values, and it has three slots.

1. Data validation — confirms that values match rules decided in advance. Does the email field contain an at sign, is the payment amount non-negative, does the plan field hold one of the three agreed values and nothing else. These conditions get written down once and run every day, and rows that fail are held aside for a person to look at rather than passed through.
2. Freshness — measures how recent the data behind a number actually is. A table that is supposed to refresh hourly but has not moved in nine hours is not showing a wrong number, it is showing a late one, and decisions made from it are decisions made about yesterday. What this slot settles is how many hours of delay each table is allowed before somebody hears about it.
3. Completeness — checks that the rows and fields that should be there are actually filled. A table that normally receives 3,000 rows a day and received 1,900 yesterday is missing something. A payment country field that has been 12 percent blank since last week quietly distorts every calculation that splits by country.

The three catch failures pointing in different directions. Validation catches values that are strange, freshness catches values that are late, completeness catches values that are short. Wire up only one and the other two directions stay wide open — a table where every rule passes and an entire day of rows never arrived is the standard example.

Tracing deals with where a value comes from and where it sits, and it has two slots.

1. Data lineage — the path a value travelled from its original source, through each transformation, to the screen showing it now. When revenue on a dashboard looks wrong and that path is written down, you can walk backwards until you find the step where it bent. The same map works forwards: before changing one source, you can count how many tables and screens depend on it.
2. Data catalog — a list of what data the company has, where it sits, and what each field means. It exists so that when four tables have similar names, nobody has to ask the person who built them which one to use. When that person leaves, an uncatalogued table becomes a table nobody can use. What gets recorded is more than a name — the meaning of each field, its owner, and when it last updated. If a field called active accounts doesn't say whether it means logged in or created a document, two fields with the same name end up used for two different things.

Stewardship sets the rules for use, and it has three slots.

1. Data governance — the system that says who is responsible for which data, who may open it, how long it stays and when it gets deleted. Writing one owner's name against each table is the first move here, because a table with no owner has nobody to fix it when it breaks.
2. Personally identifiable information (PII) — data that can single out one person, such as a name, phone number or address. Marking which fields qualify means different rules attach to those fields specifically: narrower access, a separate retention period, and exclusion from files that leave the company.
3. Anonymization — reworking those fields so the person behind them can no longer be recognized. Rounding a birth date to the year, or replacing an email with a value that cannot be reversed, leaves you able to tell that two rows are the same person without leaving any way to tell who that person is.

Teams usually meet the three groups in the opposite order from the one that works. Checking is what you notice first, but writing a check requires deciding what should be measured, and that decision belongs to the owner stewardship assigns. Put rules on an unowned table and the alerts fire into an empty room until somebody turns them off, which usually takes about a month.

Read in the working order, the three connect end to end. Checking tells you whether today's values are usable. Tracing tells you where to fix them when they aren't, and what else changes when you do. Stewardship decides who could put anything in or take anything out in the first place.

"Freshness alert on that table. Nothing has landed since eleven last night."

The number on screen isn't wrong, it's a day behind, so don't make the call off this table right now.

"Let me pull the lineage on that revenue figure. I want to see which step the two numbers split at."

A proposal to walk back up the path the value travelled and locate the bend, rather than argue about which final number to believe.

"That field is personally identifiable information, so I can't hand it over as is. I'll get you an anonymized copy."

Not a refusal of the request — an offer of a different shape that fits the access rules.

Here is how it went at Pagedeck. Mia saw weekly active accounts down 14 percent from the week before on the Monday dashboard and asked Theo about it. The first thing Theo opened was not the dashboard but the table behind it. For three days a single validation rule had been filtering out roughly 2,000 rows a day, and the filtered rows were being dropped rather than kept anywhere. An app release had renamed one pricing plan, while the list inside the rule still held the three old values.

Of the 14 percent, about 3 was a real decline and the rest was the filter. During those three days Mia had been looking for the cause in the new onboarding screen and had met twice with Noa about rolling it back. With no way to know the table was wrong, the screen was the only suspect available.

What the two of them changed afterwards was the procedure rather than the list. Filtered rows now get stacked somewhere instead of discarded, and if a day's filtered count runs more than five times the usual, both Mia and Theo get told. Rules will fall behind reality again, but the next time it happens it won't happen quietly.

The most common mistake is treating this as a cleanup that finishes. The first pass really does show dramatic results — delete the duplicate rows, fill the blank fields, and the numbers walking into meetings start agreeing that same week, which feels like completion. But data moves whenever the product moves. Add a plan to the checkout screen and values nobody has seen before start arriving that day. Remove a signup step and the events that used to count it disappear. A cleaned state is cleaned against one version of the product, so it drifts as soon as the product does. Leave it there and the same meeting reconvenes six months later, while the decisions made on drifting numbers in between go unexamined.

Pushing the whole thing onto the data team is the other one. The people who find and report the breakage are always from that side, so responsibility looks like it lives there too. In practice the breakage starts much further upstream. Renaming a field or adding one more allowed value is a product and engineering decision, and which tables that decision touches is invisible from where it gets made. The Pagedeck story is exactly this shape: whoever renamed the plan changed one value, and had no way of knowing that value was also written inside a validation rule. A field rename is a ten-second change in a code review and a three-day outage in a dashboard, and no part of the review surfaces the second half. Leave the structure alone and the data side spends its time patching from behind while the same category of incident repeats every quarter. Deciding what each new feature will leave behind, and in what shape, while the feature is being built costs less than repairing it later.