Technical SEO

Back when a website was a handful of HTML documents, this work did not need a name. A crawler could land on the front page, follow links, and reach everything the site had. The only thing a marketer had to worry about was the writing.

That assumption broke as sites grew. Once browsers started drawing screens with JavaScript, the first response a crawler received was often an empty shell. The words a visitor eventually reads were assembled a moment later, inside that visitor's browser, and a bot that takes only the first response away with it sees a page with nothing written on it. Add filters to a list view and every combination spawns its own address, so a site with 200 articles balloons into 20,000 addresses. Moving the help center behind a login, or rewriting the whole address scheme during a redesign, became something that happened every few months.

The symptom is not a low ranking. It is absence from the results entirely. Pagedeck lost half a year to exactly this. Ravi, the growth marketer, published twenty blog posts over three months and only four of them ever showed up in search. He assumed the writing was the problem, rewrote the titles and subheadings, and when nothing changed he wrote longer posts. Theo opened the site settings only after that. The blog index loaded the next batch of posts as you scrolled, which meant the crawler never encountered a link to anything past the first four. Three months of writing, and what decided the outcome was a setting nobody had thought to open.

This is why the discipline gets its own branch when search engine optimization (SEO) is divided into four. Keyword research picks the phrases worth chasing, on-page SEO shapes the wording inside a page to match them, and off-page SEO builds reputation elsewhere, mostly by earning links from other sites. All three deal with what a person will read. Technical SEO deals with the road a machine travels before any person gets there. Block that road and everything the other three branches produced never reaches search at all.

Six groups sit under this heading. Fetching covers how a bot finds and reads pages. Directives are the files that tell a bot where it may go. Duplication handling sorts out the same content scattered across several addresses. Comprehension marks up a page so machines can tell what its words mean. Performance covers how fast a screen arrives and how it behaves on a phone. Localization pairs up language versions. The order matters: when something upstream is blocked, nothing downstream produces a result. Adding careful machine-readable markup to a page that never gets fetched is repainting the sign on a shop whose door is locked.

Fetching is how a search engine gets hold of a page, and it splits three ways.

1. Crawling — the bot walks links and reads pages as it goes. A bot here is a program the search engine runs continuously, opening addresses and carrying copies of what it finds back home. It cannot invent those addresses; it reaches a new page only through a link somewhere. An article with no link pointing at it exists on the server and nowhere else as far as the search engine is concerned. That is the exact shape of the Pagedeck blog problem — the sixteen missing posts were all published, and none of them had a link a bot could follow.
2. Indexing — registering a fetched page in the searchable catalog. Crawling and indexing look like one motion but they are separate. A bot can visit a page and still leave it out if the content reads as thin or as a near-copy of something else. That is why the first question about a missing page is whether it was fetched or whether it was catalogued.
3. Crawl budget — how much fetching effort a bot spends on one site. Bots don't run forever; each site gets a rough daily allowance of pages. If 19,000 of a site's 20,000 addresses are filter-combination shells, the product page that matters waits weeks for its turn.

Directives are the files that spell the route out for a bot, and there are two of them.

1. XML sitemap — a file listing what pages exist, handed directly to the search engine. It covers pages a crawler might miss while following links, and it helps new posts get picked up sooner.
2. robots.txt — a file stating which paths may be fetched and which may not. It keeps admin screens and internal search results out of the catalog. One wrong line takes the whole site out of circulation, which is why it's the first file anyone opens when a site disappears from search.

Duplication handling deals with identical content living at several addresses. A tracking tag on the end of a link, a trailing slash that is sometimes there and sometimes not, an old address scheme still answering requests — any of these turns one article into four. The search engine picks one of them as the canonical version on its own, and its pick may not be the address you want people landing on.

1. Canonical tag — a marker inside the page naming which address is the real one. It funnels the credit earned by the other copies into a single destination, so a link pointing at any one of the four addresses counts toward the one you named.
2. Duplicate content — the state itself, where the same text sits at several addresses. There's no penalty attached, but the credit splits and only one copy appears in results.
3. Redirect — sending both people and bots from an old address to a new one automatically. Skip these during a redesign that changes the address scheme and years of accumulated credit ends that day.

Comprehension is about labeling what the words on a page mean so a machine can tell.

1. Structured data — machine-facing annotations declaring that this number is a price and this string is an author name, separate from what a reader sees. A machine reading plain text cannot tell whether "4.5" is a rating or a version number. The annotation says which one it is outright, in a format the engine already knows how to read.
2. Schema.org markup — the shared vocabulary search engines agreed to use for those annotations. Pages carrying it sometimes appear in results with a star rating, a price, or an application deadline attached rather than just a title and a description.

Performance is about how quickly a page reaches a person, and it's one of the few technical items search engines have openly confirmed as a ranking input.

1. Core Web Vitals — three measurements defined by Google: how long the largest element takes to paint, how long a tap waits before anything responds, and how much the layout shifts around while someone is reading. There are three because a page can arrive quickly and still be unusable: text that shows up fast but slides down the screen as an image finishes loading above it passes the first measurement and fails the third. The figures come from real visitors' browsers, not from a developer tool.
2. Mobile friendliness — whether text is readable and buttons are tappable on a small screen. Google fetches and judges using the mobile view, so a page designed only against a desktop layout leaves the bot looking at something different from what the team sees.

Localization applies only to sites that split the same content by language or region. hreflang is the annotation that makes the Korean and English versions of a page point at each other. Without it a search engine treats them as two similar documents, surfaces one and drops the other, or serves the English page to someone searching from Korea.

Nobody looks at all six at once. When search traffic breaks, the check runs top to bottom. Can the bot get in, did it catalog what it found, is the catalogued address the one you wanted, does the page's meaning come through, and does a person who opens it see something without waiting. Working on a later stage while an earlier one is blocked looks like effort and moves no numbers.

"That page is blocked in robots.txt. Rewriting it won't help."

Fetching is switched off, so no amount of editing reaches search.

"The same post is showing up at three addresses. Let's set a canonical tag and look again."

Credit is split three ways, so name one address as the real one and consolidate.

"This redesign isn't shipping without a redirect map."

The address scheme is changing, so the release is on hold until every old address is paired with a new one.

At Pagedeck this branch is shared ground between Ravi and Theo, with a clear split. Ravi is the one who notices traffic dropping and produces the list of pages that vanished. Theo is the one who traces it to a setting and fixes it. When Ravi says help-center traffic is down 40 percent since last week, Theo walks back through that week's releases. Spotting it and fixing it are two different jobs, and the distance between them is where the weeks go.

Last quarter the cause was a single release. A settings file meant for the staging server shipped along with a new feature, and that file carried one line telling crawlers to stay off the entire site. Reverting took five minutes; search results took three weeks to recover. Pagedeck now has a line on the release checklist for that file. Most of the work in this branch is less about building something new than about keeping things like this from breaking.

The most common assumption is that only engineers can touch any of it. The fixes live in config files and code, so it looks that way. But the expensive part isn't the fix, it's working out what went wrong, and the first evidence shows up in traffic numbers and search result screens. The person who knows which pages stopped appearing, and when, is usually the marketer. Left uncorrected, the assumption means whoever could spot the problem treats it as someone else's job while whoever could fix it never hears about it. Sites that sit blocked for months almost always die in that gap.

The other common belief is that this is a one-time setup. Once a sitemap exists and canonical tags are in place, it has a finished shape, so it reads as done. In practice these are settings that any release can break. Address schemes change, new list views appear, an experimental config leaks into production. A team that hasn't folded these checks into its release process finds out after traffic has halved, and by then the release that caused it is weeks old and tracing it back eats days.