Secret Management

Writing the value directly into the code was the default for a long time, because it's short and because it reliably works. The problem with it is that everybody who can open the repository then holds that value, whether they wanted it or not.

There was a day at Pagedeck when this became visible. A newly hired engineer pulled the repository on their first morning and found the production database password sitting in a config file. It was a line somebody had added in a hurry three years earlier and then forgotten about. The line got deleted and pushed, and that didn't end it: after deletion the value remained inside the commit history, still visible to anybody who checked out an earlier point.

Being hard to undo is the character of this problem. Erasing one value completely from a history means rewriting the entire history, and there is no reach at all into the machines of people who already cloned it. So the only real answer becomes revoking the value and issuing a new one. That's why the practice moved to not putting it in the code in the first place.

Under CI/CD, the system that automatically verifies code changes and carries them through to deployment, this work sits in the configuration group. Of the environment variables injected differently per environment, it handles only the ones that must not leak.

Five steps.

1. Separate what counts as a secret. A payment provider's address is configuration that can be public; the key used to send requests to that provider is a secret. The split has to be written down once rather than decided per value. Without the distinction every value ends up in the vault, and since changing one then requires a procedure, the team starts routing around the vault.
2. Put them in a dedicated store. What that store does is hold them in unreadable form and record who pulled which value when. The record is the decisive difference from a file on disk: after an incident you can narrow down what actually leaked and what didn't.
3. Inject at run time. The value is set as an environment variable when the process starts and the code reads the variable. Write the value into a file that goes into the build and it ends up inside the artifact, and artifacts get copied to many places, so there's no recalling it.
4. Grant access by role rather than by person. Staging keys can be pullable by every engineer while production keys are pullable only by the deployment pipeline. The aim is reducing the situations where a person has to hold a production key at all.
5. Rotate on a schedule. Set a term such as every 90 days and replace the value. The point is less the replacing than keeping the system in a state where replacement is possible. A key never once rotated can't be rotated when it actually leaks, because by then nobody knows who uses it or where it's read from.

The procedure for a leak gets written in advance too. Without a note saying who revokes what in which order, the first hour of the incident goes to deciding that order.

The moment to adopt it is when the first of three things appears: environments splitting into two or more, more than three people touching the code, or an external payment or data store getting attached. Before that there's one value and one person who knows it, so nothing about this is urgent yet.

The limits are clear.

1. The vault needs a key too. One credential for reaching the vault has to exist somewhere, and that one can't be protected the same way. In practice this layer gets replaced by an identity granted to a machine rather than a value a person holds. Using the identity a cloud issues to each server removes the need to write any credential down at all.
2. It's plaintext while running. Inside process memory the value is readable, so if an error prints it into a log, everything the vault did is undone. Masking specific named values in the logging layer gets set up alongside it.
3. It collides with convenience. Going through the vault on every local run is slow, and an internet outage stops everything. Development uses separate narrowly scoped values while production values are blocked from reaching anybody's laptop.

It overlaps visually with encryption, turning content into something unreadable, but the question each handles is different. Encryption decides what gets transformed and how; secret management decides where the key lives and who pulls it when. However well the encryption is done, a key written into the code puts the lock and the key in one envelope.

"That key is in the repository. Deleting it isn't enough — we revoke and reissue."

Deleting the line doesn't make the value go away.

"Let's make production keys pullable only by the pipeline. No person needs to hold one."

Proposing that access be granted by role instead of by person.

"Tokens are showing up in the error logs. We need masking."

Naming one of the routes by which values escape the vault anyway.

At Pagedeck, Theo builds the store and Grace approves the revocation procedure. Whether to pause production when a key leaks is a business call rather than a technical one, so Grace decides it, and that standard was written down before any incident. A leaked payment key gets revoked immediately; a leaked read-only query key gets rotated with the next deployment.

The most common position is that adding the config file to the ignore list makes things safe. Once the file stops going up, the job looks finished. But an ignore list only stops what would be committed from here on. A value committed even once remains in the history, and that history has been copied to the machine of everybody who cloned the repository. For a value already committed, the answer isn't exclusion but revocation. Reverse the order and you tidy up the list while leaving a live key in place for months.

The other frequent belief is that small teams don't need this, since a vault looks excessive for a product three people are building. But the need is set by the risk in the values rather than by headcount. A product three people built can move somebody else's money through one key the moment it starts taking customer payments. Adopting it later also costs far more than starting with it. Pulling a value out of twenty places in the code, revoking all of it and reissuing takes days of work, and deployments stop for those days.