MLOps

Building a model and keeping a model running are two different jobs, and for a long time they were treated as one. An analyst would spend two weeks on a laptop producing something with 92 percent accuracy, and that looked like the finish line. It was closer to the starting line. Getting that model into the product meant rewriting the code so it could run on a server, wiring up a way to feed it each day's new records, and carrying the predictions back out to the screen. Two weeks to build, three months to ship.

What happened after shipping was worse. A model does not keep performing the same way just because nobody touched it. With nothing watching, accuracy could slide for half a year and the first signal would be a rise in support tickets.

So planning meetings kept landing in the same spot. Theo would ask what the model was scoring right now and nobody could answer. The last measurement was four months old, no record said which date range the training data had covered, and re-measuring meant half a day of someone's time.

MLOps applies to models what teams already did for code. On the code side, continuous integration and delivery (CI/CD) — automatically checking every change and carrying the ones that pass all the way to deployment — was already standard. Do the same for models, stringing training, validation, deployment, and monitoring into one line, and the process runs whether or not anyone remembers it.

The loop has six steps, and the last one feeds back into the first.

1. Collect the data and pin down its cutoff. Training data is the full set of records used to teach the model. Store it separately with a label saying which date it runs through — rebuild the same model without the same data and you don't get the same result.
2. Wrap training in a single run. Everything from the code that extracts features, the numbers the model actually looks at, through training itself belongs in one script. It has to run on a company server, not on an analyst's laptop.
3. Make validation a gate. A newly trained model that scores below the bar never reaches the next step. Pagedeck blocks deployment automatically when recall on the spam-detection model drops under 0.80 — meaning eight out of every ten real spam accounts have to be caught. Precision and recall are the two halves of that measurement: how much of what you flagged was right, and how much of what was there you found.
4. Deploy gradually. A model that clears the gate goes to 5 percent of incoming requests first, not to everyone. Watch it for a day, widen it if nothing looks wrong, roll back to the previous model if something does.
5. Monitor both sides. Track the predictions, but also measure daily whether the incoming inputs still look like the ones used in training. Model drift — a model that used to be accurate going dull as the world around it changes — usually shows up on the input side first.
6. Retrain and loop back. When monitoring crosses the threshold, step one runs again on recent data. Pagedeck turns this loop every two weeks.

The setup earns its cost when a model is producing predictions inside the product every day and users feel it immediately when those predictions are wrong. Pagedeck's spam-account check is one: it judges 4,000 accounts a day, and blocking a legitimate team means that team can't log in, so a drop in quality has to surface the same day rather than six months later. It also earns its cost once there are several models. One model fits in somebody's memory; five models mean nobody is sure which was last retrained or on what.

The math flips for a single model that runs once a quarter to fill in a number on a report. Standing up the process takes months, and that model takes an analyst half a day to rerun by hand.

Two limits are worth knowing. First, none of it works without a data pipeline underneath — the flow that moves raw records into an analyzable shape on a schedule. If that flow doesn't land at the same time every day, automatic retraining trains on yesterday's incomplete data and ships the result this morning. Second, automation doesn't make judgments. A person decides where to draw the 0.80 recall line, and if the line is drawn in the wrong place the system will faithfully ship a bad model.

"Performance has been sliding for three weeks. Why didn't the retraining pipeline fire?"

Monitoring is in place, but the signal never got connected to retraining.

"Before we had MLOps, a single deployment took two weeks."

Using the before-and-after time difference as the argument for the investment.

"First tell me which data cutoff the model currently in production was trained on."

A question about whether records exist. Not being able to answer it on the spot means the practice isn't in place yet.

At Pagedeck the spam-account model was the first thing put on this loop. Before that, Theo retrained it by hand roughly once a month, and whenever other work piled up he'd skip two or three months. If spam tactics shifted during a skipped stretch, legitimate accounts got blocked, and the news arrived as a complaint email to Grace.

Afterward the order reversed. An alert reaches Theo the morning recall falls under the line, retraining runs that day, and the new model sits at 5 percent for a day before widening. Mia only decides how tight the threshold should be. The amount of work didn't shrink so much as the amount people had to keep in their heads.

The most common mistake is thinking MLOps arrives with a tool. Most introductions are written around tool names, so it reads that way. But what makes the practice work is the pass-or-fail rule and the loop back, not the software. Install tools without deciding what score blocks a deployment and what event triggers retraining, and you end up with a tidy history of runs while a declining model still goes unnoticed. Decide the rules first and a few scripts plus one alert are enough to start.

Believing the job ends at deployment is the other one. Ordinary code behaves today the way it behaved yesterday unless someone changes it, and that instinct carries over. Models don't work that way: when the data outside shifts, predictions start missing even though the code is untouched. Left alone, quality erodes with no alarm anywhere, and the news arrives through the slowest channel there is — customers writing in.