Data Lake

For a long time there was one way to hold data for analysis. You decided in advance which columns existed and what went in them, and only data reshaped to fit was accepted — a data warehouse. Payment records become four columns: date, account, plan, amount. Anything that made it in was ready to query, which suited the people doing the analysis.

The trouble was how much didn't fit. Log files the server writes one line at a time, images customers upload, responses from outside services whose shape differs per vendor — none of that has a place in a four-column table. Building a new structure means deciding first what you intend to analyze, and when that answer isn't known yet, the data gets waved through instead of stored.

Pagedeck ran into it directly. Mia wanted to see the path people take when they start a document and abandon it, so she went looking for the editor's interaction logs, and found they were kept on the server for 30 days and then discarded. Where three years should have been, there was a month. Nothing had been thrown away for being useless; it was thrown away because no one had settled on a structure to receive it.

Cheap storage is what made the other answer practical: take everything now and decide the structure later.

It lets you keep data before you know what you'll analyze, because the structure is applied on the way out rather than on the way in. Three consequences follow from that one shift.

1. Formats that don't match anything still get stored. Plain-text log files, support conversations, images and audio, external responses with vendor-specific fields — all of it lands.
2. Data with no current use stops being discarded. Six months later, when someone asks why customers churn, whether the history exists decides whether the question is answerable at all. Past data cannot be manufactured after the fact.
3. Collection doesn't break when a source changes. Add a field to an external service's response and a fixed-schema load fails, while a lake takes the new shape as it is. The change gets noticed when someone reads the data.

Pagedeck's editor logs show the pattern. Nobody decided in advance which interactions were worth keeping, and six months later, while reworking document sharing, the team needed a count of people who opened the share dialog and backed out. That interaction had never been logged deliberately — it was simply sitting in the pile.

Storage cost is what makes the trade sane. Holding a year of raw logs often costs less per month than a single engineer spends on one afternoon rebuilding data that was never kept.

The bill comes due on the read side. The interpretation skipped at write time didn't disappear; it moved to the reader. Every time someone opens the same file they work out again which field is the account id and which time zone the timestamps use, and five people working that out separately produce five interpretations.

Three options serve the same purpose, and the choice turns on how much you already know about what you'll analyze.

1. Keep only a warehouse. If the data is your own product's signups and payments and the columns are stable, a lake adds a second place to maintain and nothing else. At ten people, building the lake first is usually premature.
2. Leave the data at the source and pull it when needed. If an external service will hand you history on demand, copying it is wasted effort. Most services cap how far back they'll go, though, so a 90-day query window becomes your retention period whether you chose it or not.
3. Run a lake and a warehouse together. This is the common arrangement. Everything raw goes into the lake; the parts used often get cleaned up and loaded into the warehouse. That ordering is called ELT — extract, load, then transform — and deferring the transform means a bad cleaning rule can be rerun from the original. The whole flow is a data pipeline, and deciding what belongs at which stage is most of the design work.

"Let's load it into the lake for now and decide next quarter whether it goes to the warehouse."

Keep the record even though nobody is analyzing it yet.

"That figure came straight out of the lake, so it isn't cleaned."

A warning not to take the number at face value — duplicates and failed requests may still be in it.

"We have the lake but no catalog, so nobody knows what's in there."

Data accumulated without any record of what it is, which makes it unusable.

Pagedeck built its lake right after the Series A. Theo set up a daily copy of server logs, the payment provider's responses, and conversations from the support tool into one place. For the first three months, nobody read any of it.

The value showed up half a year later. Grace asked why renewals among enterprise customers had dropped, and Mia pulled a year of support conversations and grouped them by plan. The grouping she invented that day was nothing anyone had in mind when the logging started, and without the history the question would have had no route to an answer.

The most common belief is that anything stored will eventually get used. With storage this cheap, keeping everything looks like the sensible default. But without a record of what's inside and what each field means, six months on you're guessing from folder names, and once the person who built it leaves, even the guesses run out. Data that piles up unused like this is called a data swamp, and what prevents one isn't the storage technology but the habit of writing down what went in.

The other mistake is assuming a lake makes analysis immediately possible. Having data and being able to use it sound like the same thing. In practice, raw records need duplicates removed, time zones aligned, and multiple identifiers for the same account merged before anything can be counted. When no one owns that step, each analyst cleans the data their own way, and the same month's revenue shows up three different ways in one meeting.