Data and Analytics

There was a time when the only evidence available for a product decision was whoever happened to be in the room. Someone would talk to three users, come back saying a feature was confusing, and that became the decision. Someone else would talk to three different users, come back saying the opposite, and the decision flipped. Neither person was lying. Neither one was describing the whole picture either, and there was no way to tell how much of the picture each of them had.

It wasn't that numbers didn't exist. Servers were writing access logs, the billing system was recording every transaction, and support tools were storing every conversation. The trouble was that pulling a number out produced a different answer depending on who pulled it. Marketing would say the product had 120,000 users this month, engineering would say 40,000, and the next half hour of the meeting would be spent arguing about whose figure was right instead of what to build. One side was counting anyone who loaded a page and the other was counting people who logged in. Both were correct. They were using the same word for two different things, and nobody had ever written down which one the company meant.

Getting the number at all was expensive too. A request to see last quarter's enterprise revenue broken out by region would take an engineer two days of writing one-off queries. The next week a slightly different version of the same request arrived and took another two days. Nobody checked whether the resulting spreadsheet was right, because checking it would have meant repeating the two days.

Data and analytics is the field that grew out of those three problems being treated as one problem. Agree in advance on what a word like "active user" means, record the behavior so it can be counted later, pull the scattered records into one place and keep them trustworthy, and read meaning out of them. Done separately, each of those steps quietly breaks the ones around it. Done as a single chain, the number that lands in a meeting has a path back to where it came from.

The work under this heading splits eight ways: who does it (roles), how behavior becomes a record (tracking), how records get moved and maintained (processing), how a team decides what to count (metric design), how meaning gets pulled out of what accumulates (analysis), how cause gets established (experimentation), how results reach the rest of the company (delivery), and how data becomes prediction (modeling).

Roles divide the people. There are five seats, and what separates them is which part of the chain each one owns.

1. Data analyst — digs through data that already exists to answer a question. Someone asks where churn increased last quarter, and this is the seat that comes back with a number and an explanation of what it covers.
2. Data scientist — builds explanations and predictions using statistics and models. Estimating which accounts are likely to leave within three months, based on how past accounts behaved before they left, sits here.
3. Analytics engineer — prepares data so an analyst can use it directly. Joining billing, signup, and login records into a single table keyed by account means the same calculation doesn't get rewritten from scratch every time somebody needs it.
4. ML engineer — takes a finished model and makes it run inside the live product every day. Building a model and making that model survive several hundred requests a second are different skills, which is why the seat splits off.
5. BI analyst — shares the current state of the business through dashboards and recurring reports. This is the seat that turns a question an analyst answers once into a question that answers itself every Monday.

At a small company one person holds all five. As headcount grows, the first seat to separate is usually the analytics engineer, because once three analysts each calculate revenue their own way, three different revenue figures show up in the same meeting.

Six of the remaining seven form a single line. Tracking feeds processing, processing produces metrics, analysis and experimentation attach meaning, and delivery puts the result in front of people. Weakness early in that line cannot be fixed by sophistication later in it.

Tracking is where behavior becomes a record. Event tracking writes one line every time someone clicks a button or opens a screen: who, when, what. The defining property of this branch is that it doesn't work backward. To know how many people pressed the share button last month, the recording had to already be running last month. Instrumenting it today starts the count today. That is why deciding what to record belongs to the conversation where a feature gets designed, not to a cleanup pass afterward.

Processing turns raw records into something usable, and it splits three ways. A data pipeline pulls data out of scattered sources, reshapes it into a consistent form, and moves it to one destination. Payments live in the billing system, sessions live in server logs, support conversations live in a help desk tool, and left where they are they cannot be looked at together. An analytical data store is where the collected version lands. It stays separate from the database the product runs on because a single aggregate query scanning two years of history should not be allowed to slow down the screen a customer is looking at. Data quality is the work of keeping the values inside trustworthy. A few rows with an empty amount column, or one customer stored twice under two identifiers, is enough to make every calculation built on top slightly wrong, and slightly wrong is harder to catch than obviously broken.

Metric design decides what gets counted. A metric is a number defined ahead of time so the current state can be judged against it, and most of the effort in this branch goes into agreement rather than arithmetic. Counting an active user as anyone who opened the app gives 120,000 for a month; counting anyone who created at least one document gives 40,000. Which definition is right depends on what the product is for, and until somebody writes the choice down, both numbers keep showing up.

Analysis extracts meaning from what has accumulated. Analysis methods are the recurring shapes people reach for: grouping users by the month they signed up and following each group over time, or counting how many people survive each step between signup and payment. Reading those shapes correctly depends on statistics — knowing when the average misleads and the median doesn't, or whether a three percent gap across a sample of 200 people is a gap at all or just noise wearing a number.

Experimentation establishes cause. An experiment splits users at random, shows the change to one half, and compares what happens. Analysis can show that two things moved together, but it cannot say which one moved the other. Knowing that people who create a document on their first day tend to stick around doesn't tell you whether pushing new users toward creating one would make them stick around. Splitting the population and trying it is what settles that.

Delivery puts results where the company can act on them. A dashboard collects the important metrics onto one screen so the same question doesn't get re-answered by hand every week. Whether it's a good one is judged by whether anyone does something different after looking at it, not by how many numbers fit on the page. Numbers nobody acts on are decoration with a refresh schedule.

Modeling turns data into prediction. Machine learning has a system find patterns in past records instead of having a person write the rules out, and it gets used for things like filtering spam documents or flagging accounts that look likely to cancel. This branch depends entirely on the ones before it, because the training material is whatever the earlier steps collected and cleaned.

Because the eight are chained, a suspicious number on a dashboard is rarely a broken formula. Check whether an event stopped firing, whether last night's pipeline run failed, and whether someone changed the metric definition last week — in that order.

"Where did this number come from? It doesn't match the dashboard."

The same metric name is producing two different values, so the definition has to be settled before the discussion continues. Until that happens the meeting is about arithmetic rather than about the decision that prompted it.

"Let's settle it with an experiment. Turn it on for half and give it two weeks."

Rather than arguing from opinion, split the users and compare the outcomes.

"That event was never instrumented, so we can't see last month."

Behavior that wasn't recorded can't be counted after the fact. Instrumenting the event today starts a count that will be readable a month from now, and last month stays permanently blank no matter who is asked to go find it.

At Pagedeck, Mia starts the week by opening the dashboard on Monday morning. When signups are up but the share of new users who create a first document has dropped, she asks Theo what changed in that step. If the day the new signup screen shipped lines up with the day the number bent, the rest is confirmation rather than analysis.

Ravi uses the same screen from a different angle. An experiment that sent two versions of the welcome email subject line wraps up this week: the new subject gets opened more often, but the share of people who go on to pay is unchanged. Whether to keep the new subject is a call he and Mia make together.

The most common mistake is believing that collecting enough data will eventually produce answers. Storage is cheap and keeping something has no visible downside, so the belief is easy to arrive at. But answers come from the sharpness of the question, not the volume of the records. A company that accumulates without deciding what it wants to know ends up with three years of history and no ability to say how many active users it had last month. Meanwhile the pipelines multiply and hundreds of unused tables pile up, so when a question finally does matter, the first obstacle is figuring out which table to believe. A cleanup gets proposed roughly once a year, nobody can say which of the several hundred tables still feeds a live dashboard, and the pile stays where it is.

Treating analysis as something the data team owns alone is the other frequent mistake. Working with numbers takes skill, so handing it to specialists looks efficient. The part that doesn't transfer is deciding what to count, which has to come from the people responsible for the product or the campaign. An analyst answers questions; an analyst cannot invent the question that matters to someone else's roadmap. When requests arrive without one, they arrive as "can you pull some data for me," the resulting table gets opened once and closed, and the team ends up permanently busy while no decision anywhere changes.