Sitemap and robots.txt for WordPress and your own site

A sitemap is a list of the addresses you want in search engines. A robots.txt tells search engines which addresses they may not visit, and where your sitemap is. Here's how to check both in WordPress and on a site you build yourself.

WordPress

Since version 5.5, WordPress creates a sitemap and a robots.txt itself:

• The sitemap is at https://yourdomain.com/wp-sitemap.xml.
• The robots.txt is virtual: there's no file in your web folder, but WordPress shows it at https://yourdomain.com/robots.txt, with a reference to the sitemap.
• Do you use an SEO plugin, such as Yoast SEO or Rank Math? It often creates its own sitemap, for example sitemap_index.xml. You'll find the address in the plugin's settings. Submit that address to Google and Bing.

A standard WordPress robots.txt looks roughly like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/wp-sitemap.xml

Is WordPress blocking search engines?

1. Log in to WordPress and go to Settings and then Reading.
2. Under search engine visibility, look at the checkbox Discourage search engines from indexing this site.
3. This checkbox must be unticked. Click Save Changes.

If the checkbox is ticked, WordPress asks search engines to skip your entire site. It's often ticked while building the site and then forgotten.

Read more about findability in WordPress in Findability: the basics.

Your own robots.txt

If you put a robots.txt file in your web folder yourself, it takes precedence over the virtual robots.txt from WordPress. With a WebOké hosting package, the web folder is public_html of your domain. You can upload the file with File Manager in DirectAdmin (see Opening File Manager: the folders in your package) or with FTP.

A simple robots.txt that allows everything and points to your sitemap:

User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml

A site you build yourself

Do you build your site yourself, or with another program? Check whether that program creates a sitemap. If not, you can manage without: a small site with good links between its pages will still be found. A sitemap does help, though. Put it in your web folder and mention it in your robots.txt, as shown above.

Checking

• Open https://yourdomain.com/robots.txt. Is there a Disallow: / under User-agent: *? Then no search engine may visit your site. Remove that line once your site is finished.
• Is there a Sitemap: line? Open that address in your browser. You should see a list of addresses on your site, with https and your own domain.

Tip: robots.txt doesn't keep a page out of Google. A page that may not be visited can still appear in the search results if other sites link to it. If you really want to keep a page out of Google, use a noindex. You can read about the difference in the lesson robots.txt: what you do and don't block of the Getting found online course.