From search query to result

Module 1: How search engines work · Reading time: 10 minutes

Learning objective: After this lesson you can explain what happens between typing a search query and seeing the list of results, which three steps a search engine goes through, and which search engines you come across in Europe.

Key concepts: search engine, index, search results page (SERP), organic results, ads, crawling, indexing, ranking, Google, Bing, search engines that use Bing's results, AI answers, the site: search operator

A search engine doesn't search the whole internet

When you type "bloemist amersfoort" (florist amersfoort) into Google, the list of results appears within half a second. In that half second, Google doesn't visit every website in the world. That would take far too long. Google searches its own index: a huge, pre-built database of pages Google already knows, with details for each page such as what's on it, which language it's in, what it links to and much more.

That immediately explains a question many new website owners ask: "My site has been online for a week, so why can't I find it?" The answer is almost always: it isn't in the index yet (or not completely). A search engine can only show what it has found and stored before.

The three steps

Google itself describes three steps. Bing works in broadly the same way. Not every page makes it through all three.

1. Crawling. A program, the crawler (Google's is called Googlebot, Bing's is called Bingbot), fetches pages. It finds new addresses mainly through links on pages it already knows, and through a list you provide: the sitemap. Lesson 1.2 covers this.
2. Indexing. The fetched page is read and understood: the text, the title, the images, the language, and whether the page is a copy of another page. The search engine then decides whether the page goes into the index. That isn't automatic: Google itself says that not every page it processes is actually included. Lesson 1.3 covers this.
3. Ranking and serving. Only when someone searches does the search engine look in the index for the pages that fit best and put them in order. Who is searching also counts: where that person is, which language they use and which device they're on. "Bike repair shop" gives different results in Amersfoort than in Antwerp. Lesson 1.4 covers this.

The useful thing about these three steps is that for any problem you can ask: in which step does it go wrong? A page that is never crawled can't get into the index. A page that isn't in the index can't reach the top. Many "SEO problems" turn out, on closer inspection, to be a problem in step 1 or 2, and those are often technical and easy to fix.

What you see on the results page

The page with results is called the SERP (search engine results page). There's more on it than ten blue links:

An important fact: ads and organic results are separate. Google states that organic results are not influenced by buying ads, and that advertisers get no preferential treatment. Nobody can pay Google to crawl a site more often or rank it higher, either. Anyone who promises you that advertising will improve your organic position is telling you something Google itself contradicts.

And those AI answers?

For many searches Google shows an AI Overview, and it has a separate AI Mode. Google says there are no extra requirements and no special optimisation is needed to be mentioned in them. A page is eligible as a source if it's indexed and allowed to appear in search results with a snippet. So the basics from this course apply here too. Google counts visits that come in through such an AI answer in the Performance report in Search Console (lesson 2.3).

Google, Bing and the rest

In the Netherlands and Belgium, as in almost every European country, most people use Google. Still, Bing is worth your time, even if you never use it yourself:

• Bing is Microsoft's search engine and is built into Windows and the Edge browser by default.
• Several other search engines use Bing's results (in part). DuckDuckGo states on its help pages that its regular links largely come from Bing. Ecosia states that its results come from Bing, Google or its own European index, depending on country and device.

If you do well in Bing, you'll also be found more easily through those search engines. In lesson 2.5 you add your site to Bing Webmaster Tools. A search engine that builds its own index does have its own crawler and its own rules. You'll see this in robots.txt (lesson 3.3), where you can set rules per crawler.

How long does it take?

Google says you usually don't need to do anything except publish your site: most sites in the results are found automatically. Signing up for Search Console and submitting a sitemap do help to speed things up and to see what's happening. About timing, Google says some changes are visible within a few hours, others only after months. So wait a few weeks before drawing conclusions about a change.

Checking whether your site is in Google

The quickest test is the site: search. Type into Google:

site:example.com

You'll then see pages from that domain that Google has in its index. If you see results, your site is in the index. If you see nothing, that's a sign to look at steps 1 and 2. Two notes:

• site: doesn't necessarily show all indexed pages, and the number is an estimate. For precise figures, use the Page indexing report in Search Console (lesson 2.4).
• With site:example.com/blog you search within part of the site, and with site:example.com bruidsboeket (bridal bouquet) you search for a word within your site.

In practice: search like a customer

Your own search results are coloured: Google knows where you are and what you searched for before. For an honest view, use a private (incognito) window and search the way a customer would, so not for your business name but for what you sell and where: "bloemist amersfoort", "bruidsboeket bestellen" (order bridal bouquet). Bloemenatelier Lotte may find herself straight away by her own name, yet not appear anywhere for "bloemist amersfoort". That gap is exactly what this course is about.

Summary

• A search engine searches its own index, not the internet live.
• Three steps: crawling (fetching), indexing (understanding and storing) and ranking (choosing the best pages for a search). Not every page makes it through all three.
• Ads and organic results are separate: paying doesn't improve your organic position.
• Google's AI answers have no extra requirements; the same basics count.
• Bing counts too: DuckDuckGo and Ecosia, among others, use Bing's results (in part).
• With site:example.com you can quickly see whether your site is in the index.

Quiz

Exercise

Exercise D1.1: Search for yourself like a customer. Open a private window. First search site: with your own domain (or with example.com if you don't have a site, and see what happens) and note whether you see results. Then think of three searches a customer would type without knowing your name, search for them in Google and in Bing, and note where your site appears (on page 1, further down, or not found) and what kind of results are at the top (ads, map, regular results, AI Overview).

Further reading

The steps in these articles apply to WebOké's control panels; other providers work in a similar way.

• Submitting your website to search engines (Google and Bing)
• Google Search Central, How Google Search works: https://developers.google.com/search/docs/fundamentals/how-search-works
• Google, AI features and your website: https://developers.google.com/search/docs/appearance/ai-features
• Google, How results are ranked (ads): https://www.google.com/search/howsearchworks/how-search-works/ranking-results/