Payload

The simplest way for two programs to pass values is to append them to the address. Put a question mark at the end of a URL, list the items, and the receiving side reads them off. With two or three values that is a comfortable arrangement, and it is still how most read-only requests pass their filters.

It breaks down once what you are sending gets large or gains structure. Early on, Pagedeck built document saving this way, and it worked fine for short notes. Then saving started failing silently on documents longer than a couple of pages. Addresses have a length limit, and the request was being truncated partway through.

The second problem surfaced later. When Theo opened the server access logs, the body text of customer documents was sitting in them. An address gets recorded at every hop along the way, so anything carried in one was never really private.

The third is structure. Expressing that one document has three tags and each tag has a color is awkward in a single line of URL. Giving a request its own body section, to solve all three at once, is what this concept is.

A request splits into a head and a body. The head carries what is needed to deliver it — who is sending, in what format — and the body carries the content itself. The payload is that body section.

Four things get decided.

1. The format. What rules the content is written by. The most widely used is JSON, which writes data as text. A request creating a document sends the title and folder number as pairs of names and values. It is a format people can read as-is, which makes it easy to see by eye what went wrong, and every language has a parser for it.
2. Declaring the format. Which format was used goes in the head, separately. The receiver picks its parsing method from that declaration, so when the declaration and the actual content disagree the request gets rejected even though the content is fine. This is the most common error on the first day of an integration.
3. Which requests carry one. A request's nature is expressed by an HTTP method, and methods divide into those that carry a body and those that don't. Read-only requests usually send no body and put their conditions in the address instead. Requests that create or change something carry one.
4. The response side. Bodies travel on the way back too. How the request went is reported by a three-digit status code, and a body arrives alongside it. When a document is created successfully, the code saying so comes with the new document's id in the body, and the caller takes that id into its next request.

There is a size ceiling. A server or a piece of network equipment sets a line such as one megabyte and refuses anything past it. Images and attachments therefore do not go into the body directly; the file is uploaded separately and only its address goes in the body. Build without knowing the ceiling and you get a feature that works in a demo and fails on real customer data, usually silently.

The first place is the document that specifies an integration. Which fields are required and which are optional, and whether a value is a number or a string, get settled there. With that specification missing, whoever is integrating learns it by sending requests and being rejected, which is slow and puts the cost on the wrong side.

The second is finding problems. When an integration fails, the first thing Theo looks at is the bodies that were exchanged. Setting what the sender put in next to what the receiver sent back usually reveals the cause on the spot. Something does have to be filtered before logging, though. Passwords and auth tokens arrive in bodies like anything else, and they get masked in the logs. A log kept in order to find problems becomes the problem if it leaks.

The third is a webhook, where the side that had the event calls first. When a payment is approved the payment provider sends a request to our address, and what happened is in that body. The receiver has to confirm the body really came from that company, because anyone who knows the address can send something shaped the same way.

The fourth is security. Trusting a value because it arrived in a body is how incidents start. If a request that changes a plan carries both the plan name and the price, and the server records the price as given, a request with the price edited to zero goes through. Values get re-checked on the receiving side, since nothing in a body is more trustworthy than whoever sent it.

The fifth is changing a specification. Drop a field or rename it and every existing integration stops, so the move is to add the new field and keep emitting the old one for a while. Miss that distinction and you have shipped a breaking change.

"Can you dump the payload and send it over? We need to see what's actually going out."

A request to check the real content of the request body.

"It's returning 400 because the format declaration doesn't match."

The body is fine and the header describing it is not.

"Don't put the file in the payload. Upload it separately and pass the address."

Restructure so the size ceiling is not hit.

At Pagedeck, Theo writes out the list of body fields before anything else when settling an integration spec. He aligns that list with Mia before screens get drawn, because a mismatch between the values shown on screen and the fields the server accepts means fixing both sides later.

The most common assumption is that a payload is JSON. Most bodies encountered in practice use that format, so it is a natural mistake. But the body is a slot, independent of format. Uploading a file puts the file's bytes there, and on links between internal services a format people cannot read gets carried instead. Treat body and format as one thing and you will not know where to look when an integration uses a different one.

The second is that packing more into one request is better. Fewer requests sounds faster. But as a body grows, three things grow with it: the risk of hitting the ceiling, the time the receiver spends processing, and what is lost on failure. Send 500 documents in one request and a single bad record can get the whole thing rejected, leaving nobody sure what was processed. Five requests of 100 look slower and are far easier to recover from.