Fine-tuning

A large language model (LLM) is a program that has read an enormous amount of text in advance and picks what word comes next. Because it came out of whatever is lying around on the internet, its answers resemble the internet's average. Product features rarely want the average. A company has its own scheme for classifying support inquiries, and its own rules about how long and how formal the sentences that go out to customers should be.

The first attempt is always to write those rules into the instructions. Shaping the instructions given to a model is called prompt engineering, and the more rules there are, the thicker that text gets. Theo's instructions for Pagedeck's automatic inquiry classifier ended up over forty lines long: twelve category names, an explanation of every boundary that is easy to confuse, and twenty worked examples.

Two problems followed. The first is that those forty lines ride along with every single inquiry processed. A model reads text in pieces called tokens and gets billed by how many of them there are, so a longer instruction means a higher cost per item. The second is that the twenty-first shape still came out wrong. When an inquiry arrived in a form the examples hadn't covered, the model fell back on the generic categories it already knew.

Fine-tuning inverts the situation. Rather than explaining the rules in words every time, you show the model several hundred correctly handled cases and tilt the model itself in that direction. It's closer to handing a new hire the entire archive of past cases than to handing them the manual again each morning.

It runs in six steps, starting from narrowing the use to exactly one and ending in an evaluation that confirms the change.

1. Pick one thing to improve. Inquiry classification, summary formatting — one of them. Hand it several at once and there is no way to tell which got better and which got worse.
2. Collect examples. Inputs paired with their correct outputs are the training data. Tasks with consistent shapes start showing an effect from a few hundred pairs; tasks involving judgment need more than a thousand.
3. Set aside 10 to 20 percent of what you collected. That becomes the evaluation set, and having the model answer problems it never trained on is the only way to separate real improvement from memorized answers.
4. Run the training. Upload the data to the interface the model's maker provides, wait a few hours, and a model of your own comes out the other side. You don't need computing hardware of your own.
5. Run the same evaluation set against the original model and the new one and compare the scores. You want a number — classification accuracy went from 71 percent to 89 percent — before moving on.
6. Attach it to the product and watch real inputs. Inquiries in shapes the evaluation set never held keep arriving, so gather the ones it gets wrong and fold them into the next round of training data.

It fits where the job is to fix a shape. Choosing one of a fixed set of categories, holding answers to a consistent length and tone, producing results in a prescribed format every time — those belong here. Rules like these are hard to write out completely in words, and showing examples gets there much faster.

It doesn't fit where the job is to install knowledge. Fine-tuning a model so it learns yesterday's internal announcement usually fails. Instead of memorizing the content exactly, the model tilts toward producing sentences with a similar flavor, which actually increases hallucination — confidently stating things that aren't true. To have a model answer from your documents, retrieval-augmented generation (RAG), which finds relevant documents when a question arrives and attaches them to the instruction, is the right tool.

There are three limits. First, it takes a stake up front. Several hundred examples have to be produced and reviewed by people, and that usually runs from a few days to a few weeks. Second, switching models means starting over. A better model arrives, but what you trained stays attached to the old one. Third, a changed rule doesn't take effect immediately. Adding one category name is a one-line edit to an instruction, while a fine-tuned model needs the data extended and the training run again.

So there is an order to it. Try shaping the instructions, try attaching source documents, and bring out fine-tuning for whatever is still left.

"Is this really a fine-tuning job? Let's try one more pass on the prompt."

A proposal to exhaust the cheap options before the expensive one.

"We've collected 300 training examples but the labels differ from person to person. We need to agree on the criteria first."

Without consistency, training won't teach the model a single standard.

"The fine-tuned model scores well on the evaluation set but drops on real inquiries."

A sign that the collected examples didn't represent the distribution of inputs that actually arrive.

Pagedeck's automatic inquiry classification walked exactly this path. Theo held out with instructions alone for two weeks, and when accuracy stopped climbing around 70 percent he proposed fine-tuning. Mia pulled up the classification records for 1,200 inquiries from the previous six months. The problem was that the labels in those records differed by whoever had handled them: one person had filed payment errors under billing, another under bugs.

So before running any training they reagreed the classification criteria and relabeled 800 records by hand. That work took four days; the training itself finished in three hours. The ratio shows where the time in fine-tuning actually goes.

The most common belief is that fine-tuning makes a model memorize company documents. "Train it on our material" sounds like "make it remember the contents," so the assumption is natural. What the model actually learns is shape and tendency, not content. Train a model on 500 internal policy documents and ask it how many holiday days you get, and it will answer with a plausible wrong number in a sentence that looks exactly like a policy document. Left alone, this is worse than the original problem, because the answers now look more convincing and the errors are harder to spot.

Skipping numerical confirmation of the effect is the other one. Read three or four outputs and the improvement feels obvious. But feelings depend on which samples you picked, and people tend to pick the ones that went well first. A model deployed without an evaluation set leaves nobody able to say, a few weeks later, which version was better — and with no evidence to justify reverting, it just stays in place.