Machine Learning

After Pagedeck opened a free plan, accounts that mass-produced advertising documents started showing up. The first thing Theo did was write conditions. Lock the account if a single document carries more than ten outbound links. Lock it if more than thirty documents are created within an hour of signup. Lock it if the title contains certain words. A month later there were sixty conditions, and six months later there were 320.

Around then the same support ticket started arriving forty times a day: "why is our team's document locked?" Opening them showed the same story each time, a new hire copying thirty meeting-note templates on their first day. Theo raised the number in condition 117 from thirty to fifty and those tickets stopped, but the following week the advertising accounts that produced fifty at a time sailed straight through. Loosen the conditions and spam leaks; tighten them and ordinary users get caught. The two took turns.

The problem was the method rather than Theo's skill. The line between a normal account and a spam account is not drawn by a link count or a document count on its own. It is made by twenty signals moving together, and some of those signals cancel each other out. No person can write down every combination, and conditions that have been written down get read off and routed around by whoever is on the other side.

Machine learning reverses the order of the work. Instead of writing conditions, you gather the judgments people already made. Hand a program 8,000 accounts that were locked and 40,000 accounts confirmed as normal, and let it find the boundary between the two groups for itself. People supply the answers and the machine produces the rules. A job that used to mean maintaining 320 conditions by hand turns into feeding in new records each month and redrawing the boundary.

The same approach covers prediction. When Mia said she wanted to know which teams would cancel within three months, there was no way to express that as a condition. Some teams taper off before they leave, and some use the product happily until the final week. But two years of records about teams that cancelled and teams that stayed were already sitting there, and pulling out what the last few months before a cancellation have in common is work a machine can do.

That is why this branch sits at the far end of data and analytics. The material that goes into training is the data the earlier branches collected and cleaned, so if the records someone decided to keep are missing, or the values disagree with each other, there is nothing to start from.

The terms under machine learning are arranged in the order you run into them while building one model and keeping it alive. How you get it to learn is approach. What you get it to answer is task. What you feed it is input. What goes wrong is problems. How you know whether it worked is evaluation. Keeping it going afterwards is operations. Six branches.

Approach divides on whether the answers are attached. There are only two, and the data you have decides which one you use rather than any preference.

1. Supervised learning — teach with data that has the answers attached. The spam judgment above belongs here. Each of the 48,000 accounts is already marked spam or not, so the program keeps nudging its boundary toward getting those marks right. The results are more accurate, at the cost of the labour of attaching the answers first.
2. Unsupervised learning — no answers, just structure. This is for when you don't even know what you are looking for. Split 50,000 Pagedeck accounts into a few groups based only on usage patterns and the teams that write nothing but meeting notes separate from the teams that write nothing but proposals. Naming the groups and reading meaning into them is left to a person.

Task divides what the model is asked to answer, and three come up constantly. Classification answers which category something falls into: spam or not, a refund question or a bug report, one of a set of boxes fixed in advance. Clustering does not fix the boxes and instead groups similar things together, which is what splitting those 50,000 accounts was. Recommendation picks out the items a particular person is likely to care about, which is the job behind deciding which template a brand-new user sees first at Pagedeck. The difference between the three is the shape of the answer. Classification returns one predetermined box, clustering returns unnamed groups, and recommendation returns a ranked list.

Input covers what you actually feed the model. Training data is the whole body of records used to teach it, and a feature is one individual input value pulled out of those records. When you say you are feeding it an account, what the machine sees is not the account itself but a bundle of numbers: days since signup, average documents per day, people invited, share of outbound links. What you choose those numbers to be changes the result heavily. Put in cumulative document count instead of daily average and every healthy team that has been around for two years gets pushed toward the spam side. Changing one feature raises the score more often than changing the model does.

Problems covers the two ways this goes wrong. Overfitting is a model that has been fitted too closely to the data it learned from. It is memorising the answers to past exam papers instead of understanding the material: every question it has seen comes out right and every new one comes out wrong. When a model scoring 98 on its training data scores 71 on real accounts, this is the first suspect. Model drift is a model that started out accurate going blunt as time passes. The cause is not the model ageing but the world moving. When the spam side changes its methods, or a new feature shifts what normal usage even looks like, the model has not changed by a single character and its performance still drops.

Evaluation covers how you judge whether it worked. There is a reason not to look at the share of correct answers alone. If 98 percent of all accounts are normal, a model that calls everything normal no matter what arrives also scores 98 percent: a model that does nothing at all collects a high mark. Precision and recall avoid that trap by splitting the number in two. One is the percentage of the things flagged as spam that really were spam, the other is the percentage of the real spam that got caught. The two push against each other. Tighten the threshold and nearly everything caught is genuine while more spam slips by; loosen it and the reverse happens. Which side to weight is decided by the product rather than by the technique. If a wrongly locked account turns into a cancellation, the team chooses to let some spam through.

Operations covers keeping a finished model running. MLOps is the setup that ties training, deployment, and monitoring together so they run without anyone pushing them along. Training something on a laptop once and getting a good score is a different skill from having that model absorb hundreds of thousands of cases a day and raise an alert when its performance sags. Handle the other five branches well and, without this one, the model goes quietly blunt about six months later with nobody watching.

The six branches do not finish in a straight line; they come back around. A drop in performance sends you to collect fresh training data from recent records, the evaluation scores send you to change a feature, and the retrained model goes back into operation. It resembles equipment that needs constant attention more than an object you build once.

"Rules can't catch this one. Before we go the machine learning route, we need to know how much labelled data we actually have."

Without enough cases that a person already judged, there is nothing to start from.

"The model is deployed, but precision is 0.6, so we can't auto-block. Let's route it to the review queue for now."

Four out of every ten accounts it flags are innocent, so the proposal is to keep a person in the loop.

"Performance has dropped since last month. The model is the same, but the documents coming in are different."

This is not a model to fix but a signal that it is time to retrain on recent data.

The day Pagedeck first deployed its spam model, the number Theo brought to the meeting was 94 percent correct. What Mia asked about was not that number but how many healthy accounts were being locked by mistake each day. Counting them gave twelve, and half of those were paying teams. So for the first month nothing was locked automatically and flagged accounts only went onto a list for a person to read.

Three months later a different pair of numbers came to the meeting. Wrongful locks were down to two a day, and the time Theo spent adjusting conditions had fallen from six hours a week to one. What Grace asked about was neither the model's score nor its precision, but those two things: whether customer tickets had gone down, and how much of the work people used to do by hand had disappeared.

The most common mistake is treating machine learning as always better than rules. You tend to meet it first in a scene where it solves something rules could not, which reads that way. But it only works when there are past records. If there are only a few hundred judged cases, or the feature is brand new so no records exist at all, rules are the better answer. The same holds wherever the answer has to be explained. Asked why an account was locked, a rule can say "the title contained this word," and a model usually cannot. Leave this belief in place and a team spends two months on something ten lines of rules would have settled, and cannot reverse course when the result comes out worse than the rules, because people have already been assigned to it.

A second belief is that enough data makes it work on its own. Everyone hears that training needs a lot of data, so quantity sounds like the whole story. What decides the outcome is the quality of the answers more than the volume. If 1,500 of Pagedeck's 8,000 locked accounts were locked by mistake, the model learns those mistakes along with everything else, and what gets built is a machine that repeats a human error automatically. Hold on to this belief and the team reaches for more data every time the score stalls, while never touching the job of finding the records whose answers are wrong. That is why a step goes in before training starts: pull a few hundred labelled records and have a person check them again.