Module 1 · 1.2

English

Nederlands English Deutsch Français Español Italiano

Crawling: how a search engine finds your site

Module 1: How search engines work · Reading time: 11 minutes

Learning objective: After this lesson you can describe how a crawler discovers new pages, which status codes it comes across and what they mean, and what stops a crawler.

Key concepts: crawler (Googlebot, Bingbot), following links, sitemap, HTTP status code (200, 301, 302, 404, 410, 429, 5xx), redirect, crawl budget, JavaScript and rendering, orphan page, robots.txt

What a crawler is

A crawler (also called a spider or bot) is a program that fetches web pages, just as your browser does, but automatically and on a huge scale. Google has Googlebot, Bing has Bingbot. They request an address, get the HTML back, read which links it contains and add those new addresses to their list to fetch later.

Google says that, on average, Googlebot doesn't visit a site more than once every few seconds, and that most requests come from the smartphone version of Googlebot. So Google mainly looks at your site the way a phone sees it. Lesson 4.2 comes back to this.

A crawler can also pretend to be Googlebot: the name (the user agent) is easy to fake. Google recommends checking a suspicious visit with a reverse DNS lookup of the IP address. That's useful if you see "Googlebot" doing strange things in your server log.

How a crawler finds new pages

Google mentions three ways:

Revisiting pages it already knows. That's where it spots changes and new links.
Following links. A link from a page Google already knows to a new page is the most important way to get found. That applies to links within your own site (your menu, a link in a piece of text) and to links from other sites.
A list you provide: the sitemap. A sitemap is a file listing the addresses of your pages. You submit it in Search Console or add it to robots.txt (lesson 3.4). According to Google, a sitemap isn't required, but it is useful, especially for new sites with few links so far, large sites and pages buried deep in the site.

Links have to be real links

Google can only follow a link if it's an ordinary <a> element with an href:

<a href="/bruidsboeketten">Bruidsboeketten</a>          <!-- good -->
<span onclick="ga('/bruidsboeketten')">Bruidsboeketten</span>   <!-- not a link for Google -->

A button that only jumps to another page using JavaScript isn't a link for a crawler. A page that isn't linked from anywhere is called an orphan page. A crawler can only find it through the sitemap, and visitors struggle to find it too. So a new page should always be linked from somewhere in your menu or in a piece of text.

JavaScript: fetched first, run later

Many modern sites build the page with JavaScript in the browser. Google can handle that: it runs JavaScript with a recent version of Chromium, the basis of the Chrome browser. This is called rendering. Google describes it as three phases for such pages: crawling, rendering and indexing. Pages with status 200 are put in a queue to be rendered.

Two consequences:

Whatever only appears on the page after JavaScript runs, Google sees later than what is directly in the HTML.
Not every crawler runs JavaScript. Google says so itself. Text, titles and links placed directly in the HTML are therefore the safest option for all search engines. Google calls HTML generated on the server "still a good idea".

The automatic checks in this course read the HTML as the server sends it, without JavaScript. What you see in your browser may differ from that.

Status codes: what the server replies

With every page, the server sends a status code. You don't see it on screen, but a crawler reads it first:

Code	Meaning	What Google does with it
200	OK, here's the page	Content moves on to indexing (inclusion isn't guaranteed)
301 / 308	Permanently moved to another address	Strong signal that the new address is the right one
302 / 307	Temporarily somewhere else	Weak signal; the old address often stays the main address
404 / 410	Doesn't exist (any more)	An address that was indexed before is removed from the index; Google visits it less and less often
429	Too many requests	Google temporarily slows down its crawling
5xx	Error on the server	Google temporarily slows down its crawling; if an address keeps returning a server error, it eventually drops out of the index

Watch out for the soft 404: a page that shows "not found" but returns status 200. To the crawler it looks like a real page, to the visitor it doesn't. Search Console reports these pages separately. So a deleted page should return a real 404 or 410, or redirect with a 301 to a replacement.

Google does follow a redirect chain (http to https to www to the new page), up to about ten steps, but every step takes time. Lesson 4.1 shows you how to do it in one step.

What stops a crawler

Google lists these reasons why a page isn't crawled: server problems, network problems, and rules in robots.txt that block Googlebot's access. In practice you'll also see:

a site that is still in "maintenance mode" or behind a password;
in WordPress, the "Discourage search engines from indexing this site" checkbox, which adds a noindex (that's step 2, but the effect is the same: you can't be found);
security software or a firewall that blocks unknown bots with a 403 or 429;
a site that can only be navigated with JavaScript buttons.

robots.txt gets its own lesson (3.3). For now, this is enough: it's a text file at https://www.example.com/robots.txt that says which parts of your site a crawler isn't allowed to fetch.

Crawl budget: no worry for most sites

You'll read a lot online about crawl budget: how many pages Google wants to and can fetch on your site. Google says its guide on this is meant for large sites, with a rough estimate of more than a million pages, or more than ten thousand pages that change daily. If your new pages are already fetched on the day you publish them, you don't need to worry about it. For a site like that of Bloemenatelier Lotte, with thirty pages, it simply isn't an issue. An up-to-date sitemap and the occasional look at the Page indexing report are enough.

In practice: your server log

In DirectAdmin and similar panels you can view your site's access log. Search for "Googlebot" or "bingbot" and you'll see which pages they requested and which status code they got. If you see lots of 404s or 5xx errors there, you know where to start. With other providers, you'll usually find the logs under a similar name in the customer panel.

Summary

A crawler fetches pages and finds new addresses through links and through your sitemap.
Only real <a href> links are followed; a page with no links pointing to it is an orphan.
Google runs JavaScript, but later; other crawlers often don't. Important content belongs in the HTML.
The status code decides what a crawler does: 200 process, 301 move, 404/410 remove, 5xx and 429 crawl more slowly.
Crawl budget only matters for very large or very rapidly changing sites.

Quiz

Exercise

Exercise A1.2: What does a crawler see on your homepage? Enter your site's address (or www.wikipedia.org if you don't have a site of your own). Then fill in what you expect: the status code of the final page, the address you end up at after redirects, and whether it has a noindex. The program fetches the homepage the way a crawler does, without JavaScript, and shows you the redirect chain, the status code, the number of real links to your own site in the HTML, and whether the page's text can be read without JavaScript.

Cookies at WebOké

Cookie settings

Necessary

Marketing