Crawling

Publishing an article and still not seeing it in search results a month later used to be common. With no way to find the cause, the conclusion usually landed on the search engine not liking the site — a conclusion that makes nothing fixable.

The actual cause was usually further upstream: the bot had never reached the page. A bot can't invent an address; it only arrives at a new page through a link somewhere. Publish thirty articles with their addresses linked from nowhere and articles that plainly exist on the site don't exist to a search engine.

So the stages got separate names. A bot reading a page and that page being entered into the searchable list are two different things. Crawling is the name of the first, and it's the thing to check first when a page is missing from search.

The bot starts from addresses it already knows and moves to the next page through links on that page. It runs in four steps.

1. Collect starting addresses. Previously visited addresses, addresses listed in a sitemap file, and links other sites have pointed at you are the entry points.
2. Download the page and extract its links. What gets extracted is only addresses actually written in the document. A list built so that clicking changes the view without changing the address gives a bot nothing to follow.
3. Queue the newly found addresses and visit them in order. When the queue is long, addresses near the back get visited days later.
4. Decide when to come back. Pages that change often get revisited often; a page untouched for six months gets visited rarely.

Three things block it: an address linked from nowhere, a path excluded in the robots.txt file, and anything behind a login. The third is often correct to block. The first is usually a mistake.

There is a ceiling on the effort spent on any one site, called the crawl budget. With tens of thousands of worthless addresses accumulated, the bot spends its rounds on those and reaches new articles late. Addresses that don't need to appear in search should be blocked.

"This article isn't in search. Let's check whether it even got crawled."

Narrow the cause in order rather than guessing.

"The new posts aren't linked from the index page. There's no route for the bot."

Nothing links to them, so nothing can reach them.

"Filter combinations generated 40,000 addresses. Those need blocking."

Worthless addresses are eating the budget.

Ravi ran into this on the Pagedeck blog. Twenty-two new articles turned up in search not once, and when Theo looked, the blog index had been built so that paging through it never changed the address. Once the index was rebuilt to change addresses, all twenty-two appeared in search within two weeks.