Model Drift

For a long time teams treated a model's accuracy as a fixed property. You measured it once before launch, wrote the number in a document, and moved on. Nobody had touched the code since the release, so if it caught nine out of ten cases today, it seemed reasonable to expect nine out of ten next month too.

Then the support tickets arrive first. At Pagedeck, the feature that automatically locks spam accounts worked fine six months ago, and now ordinary team accounts keep getting locked, some of them asking for refunds. Theo goes through the deploy history and finds nothing changed in that window. Nobody broke anything, the results got worse anyway, and there is no change to roll back.

Model drift is the name for that awkward position. The thing that moved was not the model but everything around it. Spammers changed their approach, and in the same months Pagedeck shipped template sharing, which created legitimate teams that send hundreds of invitation emails a day. The model kept judging today's accounts against a picture of the world from six months earlier. Once the situation had a name, teams stopped treating a weakening model as a breakage to diagnose and started treating it as something that ages, which meant measuring it on a schedule instead of hunting for a culprit each time.

A model extracts rules from the data it learned on, freezes them, and pushes new inputs through those rules to produce an answer. Drift happens when the inputs arriving today drift away from what the inputs looked like when those rules were formed.

Two terms carry most of this. Training data is the historical record used to teach the model, and a feature is an individual input value pulled out of that record. For spam detection, one account arrives as a bundle of numbers: days since signup, average invitation emails per day, share of outbound links. That bundle is the entire world as far as the model is concerned.

Things come apart in two different ways.

1. The incoming values themselves change. During training, most accounts sent about three invitation emails a day. After template sharing shipped, teams sending 200 a day became ordinary. The model still reads 200 as a spam signal.
2. The relationship between the values and the correct answer changes. The distribution of numbers looks the same, but the same number now means something different. A high share of outbound links used to indicate spam; once spammers began hiding links in attachments instead of the message body, low-link spam became common.

The first kind can be spotted from the inputs alone. The second kind requires knowing the correct answers, and those take time. Whether a locked account really was spam only gets settled when a customer disputes it or a person reviews it, which lands days or weeks later. So the second kind always surfaces late.

The speed varies too. A feature launch can bend the curve in a single day, or a slow shift in user habits can take accuracy down two or three percent a month for half a year. The slow version is the more dangerous one: no individual day looks unusual, and a year later the accuracy that started at 90 percent reads 74 percent.

The term belongs to models that keep running every day, not to models built once and measured once. It appears fastest where an adversary adapts, as in spam detection; where taste moves, as in recommendations; and where seasons and the economy intrude, as in demand forecasting.

This is why operations teams attach continuous watching rather than a one-time measurement. MLOps, the practice of wiring training, deployment and monitoring together so they run without hands on them, treats drift monitoring as one of its standard items. Three things usually get watched.

1. The distribution of incoming values. If the median number of daily invitation emails climbs from 3 to 12, that signal arrives without waiting for any correct answers.
2. The distribution of the model's own answers. If the share of accounts flagged as spam goes from 0.8 percent a day to 4 percent, something outside the model changed.
3. Actual scores, measured with precision and recall. Precision is the share of flagged accounts that really were spam; recall is the share of real spam that got flagged.

When a signal shows up, the usual response is to collect fresh training data from recent records and retrain. How far back to reach is the real judgment call. Use only the last month and the model tracks the new world well but forgets rare older spam tactics; use two full years and the opposite happens. Pagedeck settled on the last six months, with a separately kept set of spam cases from the past two years mixed back in.

"Accuracy has been sliding since last month and the code hasn't changed. Feels like drift."

Look at the data coming in before looking for a bug in the model.

"This launch is going to shift the input distribution. Let's schedule a retrain up front."

Treat drift as something to plan for in the release, not something to clean up after an incident.

"Performance hasn't dropped yet, but the input monitor fired."

Early signs are visible while predictions are still holding up.

At Pagedeck, Mia opens the spam detection dashboard every Monday. It shows last week's locked accounts, how many of those were released after a customer disputed them, and the precision of the sample a person reviewed. When the released count rises two weeks running, it goes to Theo, and Theo checks the distribution of input values against last quarter before reading any code history.

Last quarter that order found the cause in half a day. Teams using template sharing were sending roughly twenty times more invitation emails than other accounts, and the model was reading that one number as spam. Rather than swapping in a different model, they pulled recent records from those accounts into the training set and retrained.

The most common reading is that the model got old and wore out. Most things people use do wear out, so the assumption is natural. But a model is stored numbers, and left alone it behaves exactly the same a year later. What changed is the data arriving at it. Hold this belief and the instinct is to swap in a better model — which then learns from the same stale training data and lands back in the same place two months later. The thing to fix is not the model but the record it learned from.

The other common reading is that steady accuracy means no drift. Performance numbers can only be computed once the correct answers are attached, and those arrive days or weeks later. There really are stretches where the inputs shifted three weeks ago and the performance chart is still flat. The accounts wrongly locked in that gap are already looking at the cancellation screen.