Prompt Engineering

A large language model (LLM) is a program that has read an enormous amount of text and picks what word plausibly comes next. It isn't looking facts up; it's choosing a likely continuation, which means the result swings hard on how you asked.

When Theo first built meeting-notes summarization at Pagedeck, the instruction was one line: "summarize the following meeting notes." The output was different every time. Some meetings came back in two lines and some in ten, and decisions sat at the same weight as passing small talk. Mia reviewed it and called it unusable.

The team's first instinct was to switch models or train one on their own material. What actually got changed was the instruction. Split into separate lines — extract only what was decided, stay under three lines, never write anything that isn't in the notes — the same model produced something usable.

Prompt engineering is that work turned into a procedure. Nothing about the model changes; only the instruction does, which makes it the cheapest and fastest lever available. So it's where an AI feature usually starts, and the more expensive options come out only after this one has been shown to fall short.

The first step isn't writing an instruction. It's writing down what counts as passing.

1. Define what an unusable result is. Whether a summary is "good" gets judged differently by different people, but "a decision is missing" is a failure and "a sentence appears that isn't in the notes" is a failure, and written down that way the judgment stops moving.
2. Write the instruction in four blocks: who to answer as, what to do, what format to return, and what not to do. Crush all four into one sentence and the later conditions get ignored.
3. Attach examples. Format is more precisely shown than described. Two or three input-output pairs pin down length, tone and the order of items better than any adjective does. Draw the examples from the hard cases rather than the clean ones; a set of tidy inputs teaches nothing about the messy notes that actually arrive.
4. Fix the output into something a machine can read. Take prose and your code has to take it apart again, and that parsing breaks the moment the model phrases a sentence slightly differently. Ask for a structure with named fields instead.
5. Make not knowing an allowed answer. Say that when the basis isn't in the material provided, the field should be left empty. Without that line the model leans toward filling blanks, and what it fills in ends up recorded as a decision nobody made.
6. Collect failures into a fixed set. Take twenty real meeting notes where the output went wrong and re-run all twenty every time the instruction changes. Fixing one case commonly breaks another, and without the set the work circles in place.
7. Keep the instruction like code. Record who changed what and when, along with the before-and-after output.

It fits where the model can already do the thing and you're shaping how it comes out: summarizing, classifying, tightening sentences, converting formats. The work runs in hours and shipping means editing an instruction, which is reason enough to try it before anything else. It's also the one lever someone outside engineering can move, which matters when the person who knows what a good summary looks like isn't the person writing the code.

Three situations don't fit.

1. When the answer needs knowledge the model doesn't have. Internal policy and your own customer data were never in front of it, and no amount of instruction-polishing produces them. Retrieval-augmented generation (RAG), which finds the relevant passage in your own material and attaches it to the instruction, covers that gap.
2. When a tone or a classification scheme has to be locked in. Rather than re-explaining it in every instruction, fine-tuning teaches it to the model once. It needs hundreds of examples, so it comes out after the first two options.
3. When the result has to be correct. An instruction shifts probabilities and cannot reach certainty. Calculating an amount or deciding a permission stays in code; the model handles the prose around it.

One more limit is the size of the space. The instruction, the material you attached and the answer all have to fit inside the context window — the total amount of text the model can hold at once. Paste twenty examples in and the meeting notes you actually care about start losing room.

"I think a prompt covers this. We don't need to go all the way to RAG."

It's inside what the model already knows, so try the cheap lever first.

"Stop taking the output as prose and fix the format. Our parsing keeps breaking."

Have it emit a structure the code can consume.

"Did you re-run the twenty? The thing we fixed last time is broken again."

Check the fixed set every time the instruction changes.

Pagedeck's meeting summarization now lives in one instruction file. Theo keeps it in the same repository as the code, and every change is followed by a run of the twenty-case set with the outputs placed side by side. Mia approves or rejects from that comparison alone.

What changed after moving to this setup is that the conversations got shorter. It used to start with establishing what exactly was wrong about a summary. Now it starts from which of the twenty cases lost what.

The most common belief is that a well-written instruction can do anything. Change one line, watch the output improve sharply, and that's a natural conclusion to draw. But this lever pulls out what the model already holds; it doesn't create what isn't there. Tell a model that has never seen your internal policy to answer according to that policy and you get sentences shaped like policy. The worse part is that the reader can't tell they're wrong. A quick check on any answer is to ask where in the material it could have come from. When the missing piece is knowledge, the move is to attach material rather than to polish wording.

The second is treating it as write-once. An instruction that runs well looks as stable as code. Yet the same instruction behaves differently a few months later — a model version moves, or the material being attached grows longer, and the output drifts. Teams running without a fixed set find out from a support ticket, at which point there's no record to trace when the behavior changed. In an agent setup, where several steps run in sequence, the drift spreads further, because each step's output is the next step's input.