XML Sitemap

A search bot can't invent addresses. It only reaches a new page by following a link from somewhere. A page nothing links to exists on the site and does not exist as far as the search engine is concerned.

Pagedeck's help centre demonstrated it. Of 200 articles, 110 could be found in search results. Theo traced it: the help centre's front page linked the twenty most popular articles and the rest were reachable only through the site's own search box. Bots don't type words into search boxes, so ninety articles went undiscovered indefinitely.

It names the pages that link-following misses, and it gets newly published pages picked up sooner.

1. It lists pages nothing links to. Write all 200 help articles into the file and crawling stops being the thing that decides which ones exist.
2. Each address carries the time it was last changed. Bots use that value to decide what to revisit, which is what gets an edited page reflected within days.
3. It becomes the baseline for what got registered. Submit the file in a search engine's webmaster tool and it reports the number of addresses submitted against the number that reached the index. Submit 200 and see 130 indexed, and finding what the remaining 70 have in common is the next task.
4. On large sites it steers the bot's limited capacity. With 20,000 addresses of which 3,000 matter, only those 3,000 go in the file.
5. Putting the wrong things in muddles the signal. Non-canonical duplicates, pages marked not to be indexed, and addresses that redirect elsewhere all make the file and the page say different things.
6. Listing a page doesn't guarantee it gets registered. The file is a request and the search engine decides. A thin page stays out of the index whether or not it's listed.

Fifty thousand addresses per file is the accepted ceiling; past that, split into several files and publish a separate index of those files. Note the file's location in robots.txt.

"110 of 200 help articles are indexed. Start with a sitemap."

The diagnosis is that too many pages aren't reachable by link.

"200 submitted, 130 indexed. We need to look at the other 70."

Use the file as the baseline for identifying what's missing.

"Addresses that redirect are still sitting in the sitemap."

The file and the pages are contradicting each other.

Theo built the sitemap, Ravi submitted it, and over three weeks the indexed count went from 110 to 186. The fourteen stragglers were three-line articles, and rather than removing them from the file the team decided to fill them out. Getting a page listed and making a page worth listing turned out to be separate jobs.