The Page indexing report and URL inspection

Module 2: Search Console, Bing and IndexNow · Reading time: 13 minutes

Learning objective: After this lesson you can use the Page indexing report to see which pages are not in the index and why, and examine a single page with URL inspection. You will also know what Request indexing, Removals and Manual actions do and don't do.

Key concepts: Page indexing report, indexed, not indexed, reasons, Validate fix, URL inspection, Test live URL, Request indexing (quota), user-declared and Google-selected canonical, View tested page, Removals (temporary), Manual actions, Security issues

The Page indexing report

In lesson 1.3 you read that not every page ends up in the index. The Page indexing report (in the menu under Indexing, Pages) shows how that works out for your site. At the top you see two numbers: Indexed and Not indexed. Below them is the table "Why pages aren't indexed", with the number of pages for each reason.

First, the most important point: the goal is not to get everything indexed. Google itself says you should make sure the canonical version of every important page is in the index, and that you shouldn't expect a hundred percent. A site with tags, filters, search pages and variants will always have pages that are rightly left out. So the question is always: is there a page among them that does belong in the index?

The reasons in plain language

Reason in the report	What is going on	A problem?
Server error (5xx)	The server returned an error when Google visited	Yes, if it persists. Check with your hosting provider
Redirect error	A redirect went round in a loop, was too long or was broken	Yes: fix the redirect (lesson 4.1)
Blocked by robots.txt	Robots.txt forbids Google from reading the page	Only if the page should be in the index (lesson 3.3)
Excluded by 'noindex' tag	The page itself asks not to be indexed	Only if that is by mistake
Soft 404	The page returns status 200 but looks like "not found" or is almost empty	Yes, if it is a real page: make it useful or return a real 404
Not found (404)	The page doesn't exist (any more)	Usually not. It is if links point to it: redirect it
Blocked due to unauthorized request (401) / Blocked due to access forbidden (403)	The page asks for a password or refuses Google	Only if the page should be public
Page with redirect	The address redirects to another address	No, that is normal; the target gets indexed
Alternate page with proper canonical tag	This page points to another page with a canonical	No, that is how it should be (lesson 3.5)
Duplicate without user-selected canonical	Google found a duplicate and chose for itself	Check whether Google chose the right one
Duplicate, Google chose different canonical than user	You pointed to page A, Google chose B	Find out why (lesson 3.5)
Discovered - currently not indexed	Google knows the address but hasn't crawled the page yet	Normal for a new site; be patient and add internal links
Crawled - currently not indexed	Google read the page but hasn't included it (yet)	Check whether the page really adds something (lesson 7.3)

There are also two warnings: Indexed, though blocked by robots.txt (Google included the address without reading the content, often because links point to it) and Page indexed without content.

Click a reason to see example addresses. Once you have fixed something, you can choose Validate fix. Google then rechecks the known addresses with that problem; according to Google this usually takes up to about two weeks. Don't keep clicking: wait until the validation has passed or failed. Fixing works without validation too, by the way; Google will notice at the next crawl.

URL inspection: one page under the microscope

At the top of Search Console there is a search bar. Paste a full address from your site there and you get URL inspection. First you see what Google has in its index: "URL is on Google" or "URL is not on Google", and under Page indexing, among other things:

when Google last crawled the page, and Crawled as (the mobile or the desktop crawler);
whether robots.txt allows crawling, and Indexing allowed? (is there a noindex?);
Page fetch: did fetching the page succeed?
the User-declared canonical (what you put in your canonical) and the Google-selected canonical (what Google chose).

This is the state in the index, not a live measurement. If you have just removed a noindex, it will still show here.

Test live URL

With Test live URL, Google fetches the page right now. That way you can check a fix immediately. With View tested page you see the HTML after JavaScript has run, the response headers, console messages and, if the test succeeded, a screenshot. That is the place to check whether your title, canonical or structured data are really there when they are added with JavaScript.

Note: a successful live test is no promise. The test doesn't look at quality, duplicate content or the choice of canonical.

Request indexing

After a live test you can choose Request indexing. Google then puts the page in the queue to be crawled. Three things to know:

There is a daily quota per property. For many pages at once, a sitemap is the way to go (lesson 2.2).
Requesting the same page several times doesn't make it any faster.
It is not a guarantee: crawling can take days to weeks, and after that Google still decides whether to include the page.

So use it for an important new page or after a fix, not as a daily routine.

You can only inspect addresses from a property you own or are a user of. Want to look at someone else's page? Use public tools such as the Rich Results Test (module 5).

Three other reports you should know

Removals. This lets you remove a page from the search results temporarily (about six months), for example if personal data was put online by mistake. The page stays in the index. To remove it permanently, make sure it returns a 404 or 410, is behind a password, or has a noindex. Robots.txt is not suitable for this: Google can't read a blocked page, so it can't see the noindex either.

Manual actions. A manual action means that someone at Google has determined that pages violate the spam policies (lesson 8.2), for example through hidden text or unnatural links. Those pages may rank lower or disappear from the results. If there is no message here, all is well. If there is: fix it on all pages and then submit a reconsideration request.

Security issues. This is where Google reports if your site appears to be hacked or is spreading malicious software. Visitors may then see a warning. That is an emergency: see the WordPress course and Safe online for how to recover.

In practice: after launching a new site

Lotte launched her new site with the AI Builder and forgot that the site was still set to "do not index". A week later, in Page indexing, she sees almost everything under "Blocked by robots.txt". She flips the switch and publishes again, inspects her homepage, sees in the live test that crawling and indexing are now allowed, and requests indexing for the homepage and the Bruidswerk (bridal flowers) page. For the rest she waits for the sitemap. In WordPress the same trap is the "Discourage search engines from indexing this site" checkbox; other website builders have something with a similar name.

Summary

Page indexing shows indexed and non-indexed pages with the reason; not everything has to be in the index.
Read each reason as a question: does this page belong in the index? If so, fix it and validate.
URL inspection shows the state in the index; Test live URL shows the page as it is now, including after JavaScript.
Request indexing has a quota, doesn't help if you repeat it and guarantees nothing.
Removals is temporary; Manual actions and Security issues should be empty.

Quiz

Exercise

Exercise D2.4: Inspect one page. Choose an important page on your site (for example the page with your services) and inspect it in Search Console. Note whether it is indexed, when it was last crawled, which canonical you declared and which one Google chose. Then run a live test, open the tested page and check whether the title is in the HTML. Finally, look in Page indexing to see which reason occurs most often and whether it includes pages that do belong in the index.

Exercise D2.4: Inspect one page

No site or business of your own? Work it out on paper or skip this exercise; only the exam counts towards the certificate.

Inspected an important page
Noted the indexing status, last crawl and both canonicals
Tested the live URL
Found the title in the tested HTML
Checked the most common reason in Page indexing

Address and indexing status

User-declared and Google-selected canonical; do they match? If not, why?

Most common reason and number; do those pages belong in the index?

Your ticks and notes only stay in this browser; we don't send them anywhere. If you're signed in, we only save in My courses that you've done the exercise.

Cookies at WebOké

Cookie settings

Necessary

Marketing