Supervised Learning

Writing rules by hand has a clear ceiling in some domains, and spam filtering is the easiest example.

It started with conditions. Certain words in the subject line route to spam; sender addresses on a list get blocked. You can grow that to two hundred conditions. Then senders change the words slightly — a period inside a word, a lookalike character — and the conditions collapse. Write two hundred more and it changes again.

The deeper problem is the case where a person can't put the rule into words at all. A human recognizes spam in three seconds, but asked how, they can't write it down as sentences. The judgment combines wording, sentence shape, and sender address, and it doesn't decompose into conditions.

Supervised learning hands that judgment over as examples rather than rules. Show fifty thousand messages marked spam and fifty thousand marked legitimate and let the model find the difference itself. People supply only the answers; nobody writes the rule. That trade is the whole bargain of the method — you give up being able to state the rule in exchange for not having to.

Five steps, and the first two consume most of the time.

1. Define what you're predicting, in one sentence. For finding customers likely to churn, churn has to be defined first — thirty days without a login, or a cancelled paid plan, produce different models.
2. Collect data with answers attached. This is the training data, and this step is both the premise of the method and its largest cost. Data usually exists without answers on it, so a person has to mark each case. Building a model that sorts support requests by type means someone writing the type onto three thousand past requests.
3. Split the data into training and validation, commonly eight to two. The split exists to separate memorizing from learning. Measuring performance on the data used for training is taking an exam after seeing the questions, and the score it produces is always flattering.
4. Train. The model adjusts the relationship between inputs and answers. Which family of methods applies depends on whether the task is classification, predicting which category something belongs to, or something closer to regression, estimating a value.
5. Measure on the validation data. Strong training performance with weak validation performance is overfitting — fitted so tightly to the training data that it's useless on anything new.

Decide in advance what performance means. Accuracy alone frequently misleads. If churn runs at 3 percent of customers, a model that answers "no churn" every time is 97 percent accurate. So you separately read how many of the actual churners it caught, and how many of the customers it flagged were actually leaving. Those two numbers trade against each other, and which one matters more depends on what you'd do with the prediction.

Three conditions make it fit: the target is well defined, enough labeled examples exist, and future data will resemble the past. The third is the one most often skipped — a model trained on the past is wrong in situations that differ from the past.

Three situations are a poor fit.

1. When answers can't be assigned. If people themselves can't agree on what a good answer is, this method has nothing to learn from. Look instead at unsupervised learning, which finds structure without answers.
2. When examples are few. You can't teach a model to predict something that happened twelve times in two years. Twelve cases produce memorization of twelve cases, not a rule, and the model will look excellent right up until the thirteenth.
3. When the rule is already clear. Attach a model to "amounts over a threshold need approval" and you've added training, validation, and operations to something one conditional statement handles.

The limits are real. The quality of the answers sets the ceiling on the result. Ask three people to sort support requests into five types and if they sort differently, a model trained on that data learns the disagreement too. So before labeling starts, the criteria get written down and a sample gets marked by several people to see how much they agree.

Ageing is the other limit. As the product and the customers change, rules learned from past data drift out of true. Shipping isn't the end — measuring performance continuously and retraining periodically comes attached, which is a standing cost rather than a project.

"We need at least three thousand labeled cases. We have four hundred marked."

The premise of the method isn't in place.

"Ninety-four percent on training and 71 on validation. That's overfitting."

Diagnosing the state from the gap between two numbers.

"Ninety-seven percent accurate and it catches almost none of the actual churners."

The wrong performance measure was chosen.

At Pagedeck, Mia and Theo tried this on support request classification — sorting past requests into five types. Two years of data gave four thousand cases, but none of them carried a type, so Sofia spent two weeks writing types onto three thousand of them.

Those two weeks were half the project. After the first two hundred, Sofia and Mia had sorted more than thirty cases differently, so the criteria got rewritten and the labeling restarted from the beginning. Had all three thousand been labeled before aligning the criteria, that disagreement would have gone straight into the model, and nothing in the training numbers would have revealed it.

A common belief is that having a lot of data is enough. Volume setting performance is only half true. What this method needs isn't data but data with answers. A hundred thousand support records with no types on them is not a starting point, and attaching answers is human time, so more data means more cost. Which is why real plans start by deciding how many cases get labeled, and that budget decides whether the method is viable at all.

The other belief is that a model that predicts well also explains why. A model that gets it right ought to tell you the cause. But what this method finds is things that move together, not causes. Discovering that customers whose login frequency dropped tend to churn doesn't tell you whether raising login frequency reduces churn. That question belongs to an experiment, and treating a prediction as an explanation is how teams end up optimizing a signal instead of a cause.