robots.txt

A crawler follows links and reads every page it can reach. Without guidance it reads places there's no point reading: the site's own search results, print versions of pages that already exist, admin screens.

Two things follow. One is irrelevant screens landing in search results — someone searches the company name and gets an internal search results page. The other is load on the server. A crawler working through thousands of URLs in a short window slows the pages real users are waiting on. Neither problem announces itself, and both tend to get found weeks later.

robots.txt is the notice posted at the door. It lists where crawling is welcome and where it isn't, and crawlers check the file before they start.

It solves having no way to say which parts not to read. One file speaks to the whole site at once.

Its location is fixed: immediately after the root address. Put it in a subfolder and no crawler will look for it. The contents use a three-part syntax — which crawler is being addressed, which paths to disallow, and where the XML sitemap listing the site's pages can be found. That last line isn't required by the syntax, and it's conventional to include it.

What trips people most often is what the file actually blocks. It blocks crawling, not visibility. A page already in the index doesn't disappear by being listed here, and if another site links to that URL the address can stay in results with no content behind it. Anything that genuinely must not be seen gets blocked by access control or carries a no-index instruction in the page itself.

It also has no enforcement. This is a convention, so some crawlers honor it and some ignore it. Well-behaved search crawlers do; scrapers built to harvest content often don't.

The largest accident runs the other way. Ship a line that disallows everything, left over from development, and the whole site drops out of search.

"The site vanished from search. Let's check robots.txt first."

Look for a leftover line blocking everything before anything else.

"Blocking the admin pages is an access control job, not this file's."

Preventing crawling and preventing access are different jobs.

"Let's put the sitemap address in here."

Point the crawler at the list before it starts wandering.

At Pagedeck, the site dropped out of search the day Theo shipped a new version. A blanket disallow line meant for the staging server had gone up with it.

Ravi added the file to the deploy checklist afterwards, because even once it was reverted, search results took ten days to come back. The file runs two lines and is the cheapest single point of failure on a site.