Indexing

Publishing a page and finding it doesn't show up in search is common. The first suspicion is usually that the crawler hasn't arrived, and the logs often show that it did. Visited, and absent from results.

Pagedeck's blog was in that state. Ravi published forty posts and twenty-two of them appeared in search. Opening the webmaster tool showed the other eighteen marked as discovered but not stored.

Collection and storage being separate steps is what surfaces here. A search engine doesn't store everything it crawls. It decides for itself whether a page is worth keeping, and skips the ones it judges aren't.

Three steps follow the crawl: the content gets interpreted, a decision is made about storing it, and only stored pages become candidates for results. The crawl itself is crawling; this page covers what comes after.

Four reasons cover most non-storage.

1. Several near-identical pages — when almost the same content lives at multiple addresses, the engine stores one of them as the representative. Tracking parameters in URLs commonly leave the same post existing at five addresses.
2. An instruction not to store — a noindex marker on the page keeps it out. Shipping with a marker left over from development is a frequent mistake.
3. Thin content — a page that's mostly a list with a few lines of body text either doesn't get stored or gets dropped later. Those pages are thin content.
4. Low importance — a page nothing on the site links to sits at the back of the queue. Where the crawl budget, the crawler's capacity for one site, is tight, it can stay there for months.

There are ways to ask. Submit a file listing the site's addresses, and request individual pages through the webmaster tool when they're urgent. A request doesn't guarantee storage; it only moves the page up the queue. To remove a page instead, use the noindex marker: blocking the crawler alone can leave what's already stored in place.

"The crawler came and it still isn't indexed. This isn't a crawling problem."

Separating collection from storage.

"These URLs have the same content. Let's pick one canonical address."

Near-identical pages are splitting the storage decision.

"To take it out of search you need a noindex marker. Blocking access won't remove it."

Correcting how removal works.

Ravi sorted the eighteen. Nine were tag listings with almost no body text, five were the same post at different addresses, and four had nothing linking to them. The tag listings went to noindex, the duplicates got a canonical address, and the remaining four picked up links from related posts. Six weeks later, stored posts went from twenty-two to thirty-one.