Crawl Budget

It's easy to assume that publishing a page puts it in search. Before anything can rank, though, a bot has to come and fetch it, and bots do not walk every site without limit.

That became clear when Ravi published 8,000 template gallery pages in one week. Two weeks later roughly 400 of them appeared in search. Nothing was malformed and nothing was blocked. The bot simply hadn't arrived yet.

The more frustrating part was that the bot had arrived and was walking the wrong places. It worked through tens of thousands of addresses generated by sort and filter combinations while passing over the new template pages. Crawl budget is the name for the capacity being misspent in exactly that way.

Two things together set the size of that capacity.

1. What the server can take. A bot watches response times and paces its requests so it doesn't slow the site down. When responses lag or server errors appear it backs off; while answers come back quickly it pushes harder.
2. Reasons to come. Sites that change often and sites people search for a lot get visited more. A page untouched for six months sees the gap between visits widen.

So the work is less about raising the capacity than about cutting the waste, and waste usually collects in four places.

1. Addresses that multiply through combination. Once sorting and filtering live in the URL, one list becomes hundreds of addresses. Block those in robots.txt.
2. One piece of content at several addresses. Name a single canonical address and point the rest at it.
3. Long chains of moved addresses. Redirect A to B and B to C and one page costs three requests.
4. Pages that no longer exist. Leave internal links pointing at removed addresses and the bot keeps going back for them.

To pull the bot toward what matters, collect the important addresses in a sitemap.xml and submit it. Crawling finishing doesn't put a page in search, so check indexing separately. Crawling is the bot walking through pages; indexing is registering what it found as searchable.

"It's a 300-page site. Crawl budget isn't the stage we're at."

Too small for capacity to be the bottleneck.

"Half the bot requests are going to filter URLs. Let's block them in robots.txt."

The capacity is being spent on addresses that serve no purpose.

"Bot requests dropped this week and server response times went up at the same time."

Responses slowed, so the bot throttled itself.

Pagedeck put this on the checklist after the gallery launch. Counting which addresses the bot had fetched in the server logs, Ravi found more than half were sort combinations. After blocking those and resubmitting a sitemap containing only gallery addresses, indexed pages went from 400 to 6,000 over three weeks. Not one page was edited.