Classification

Some jobs consist of making the same judgment a few hundred times a day. The Pagedeck support inbox was one of them. Around three hundred messages arrived daily, and Sofia read each one and filed it under refund, bug, how-to, or other before passing it to whoever owned that bucket. Fifteen seconds per message sounds trivial, but it ate more than an hour a day, and the dividing line moved whenever a different person did the filing.

So the team tried writing the rules down as keywords. A message containing "refund" goes to the refund bucket; a message containing "doesn't work" goes to the bug bucket. Within a month there were more than sixty rules and they were colliding with each other. "What is your refund policy" is a how-to question and kept landing in refunds, and "I paid but the screen hasn't changed" got a different answer depending on which rule fired first.

The blocking problem was clear enough. Sofia sorts those messages correctly, and she cannot write down what she is sorting on. She isn't reading one keyword — she's taking in the tone of the whole message, whether the sender has a payment history, how long the message runs, all at once.

Classification finds that standard in the records instead of asking a person to state it. Two years of inbox history holds forty thousand messages with an answer already attached to each one. Let a program adjust its own rule until it matches those answers, and a standard nobody could put into words survives as a combination of numbers.

Define the buckets, gather labeled records, train on them, score the result, and wire it into the live flow. Six steps.

1. Define the buckets. They can't overlap, and every possible item needs somewhere to land. Keep refund and payment error as separate buckets and "I got charged twice" belongs in both, which makes the answers inconsistent from the start.
2. Gather records that already carry the right answer. The body of records used for teaching is the training data, and learning from records with answers attached is supervised learning. Each bucket needs a few hundred examples, and thin buckets like "other" are the first to score badly.
3. Decide what the program gets to look at. It never sees the message itself, only a set of numbers pulled out of it, and each of those values is a feature. Message length, whether a given word appears, whether the sender is on a paid plan, whether there was a payment in the last seven days.
4. Train. Hold back about 20 percent of the records instead of using all of them, and save that slice as an exam for later. Testing on the same questions used for teaching inflates the score.
5. Score it. Check against the held-back slice, and don't stop at the share of answers it got right — split it into precision and recall. What share of the messages sent to the refund bucket were really refunds is one number; what share of the real refund messages made it to the refund bucket is the other.
6. Wire it into the live flow. A model doesn't declare one bucket — it returns a confidence score for each one, so you draw a line for where automatic filing starts. Pagedeck set that line at 0.85, which sends 70 percent of messages on automatically and leaves the rest for Sofia.

It fits when the buckets are fixed in advance, when people have been making the same call long enough to leave a record of it, and when the call will keep repeating. If you don't even know what the buckets should be called, clustering comes first, and if the judgment is needed once, a person is cheaper.

Four limits come with it.

1. Changing the buckets means starting over. Add a "feature request" bucket to the inbox and none of the forty thousand past messages carry that answer, so they have to be labeled again.
2. It handles skewed data badly. When one bucket holds 2 percent of everything, a model that ignores that bucket entirely still scores high. You have to rebalance the counts or move the decision line.
3. It can't take on judgments people disagree about. If Sofia and another agent have been filing identical messages into different buckets, the contradiction is sitting inside the training data, and the model will never score better than that contradiction allows.
4. It goes dull over time. Ship new features or watch the way customers phrase things shift, and the model is unchanged while its score falls. That is model drift, and it's why rescoring on a schedule and retraining on recent records — the MLOps side of the work — comes attached.

A score of 98 on the training data and 71 on live messages is a different problem. Suspect overfitting first: the model fitted itself too closely to the exact records it was taught on.

"We can do this as a classification problem. Two years of tickets already have answers on them."

The buckets exist and labeled records exist, so the conditions are met.

"Classification accuracy is 96 percent, but it basically never gets the 'other' bucket right."

The overall score is high and the model still can't be used on the thin buckets.

"Let's send anything below 0.85 confidence to a human."

A proposal about how much to automate — lower the line and more gets filed automatically, along with more misfiles.

At Pagedeck, Mia proposed the work and Theo built it. The first thing Mia did was not pick a model but clean up the buckets. She and Sofia took 500 past messages and refiled them into the four buckets, found that they disagreed on 14 out of every 100, and only started training after rewriting the bucket boundaries based on those cases. Four months later the score drifted down because new feature names launched in the meantime were absent from the training data, so Theo retrained on the last three months.

The most common mistake is reading a high accuracy number as "good enough to ship." It compresses everything into one figure, which makes it look comparable. That figure only means something when the buckets receive roughly even volume. Where two out of a hundred items are spam, a model that does nothing scores 98, and shipping on the strength of that number produces a report saying everything is fine while none of the two get caught. Breaking the score out per bucket exposes it immediately.

Reading the model as the decision-maker is the other one. What comes out is a confidence score per bucket, not a verdict, and a person decides where automatic handling begins. Where that line goes isn't a technical question but a question of what happens when the model is wrong. A misfiled support ticket costs an agent one forward, so the line can sit low. Locking a legitimate account as spam costs a cancellation, so the line sits far higher.