Allowing or blocking AI crawlers: robots.txt and llms.txt
AI services such as ChatGPT, Claude, Gemini and Perplexity fetch web pages to train models, to provide search answers or to answer a user's question. With robots.txt you specify which of those robots are allowed to visit your site. There is also the llms.txt proposal. This article explains which robots exist, how to allow or block them, and what llms.txt does and doesn't do. We use example.com as an example.
1. The most important AI robots
Most AI companies use separate names for training, searching and fetching on behalf of a user. This lets you block training, for example, while still appearing in search answers.
| Name in robots.txt | From | Purpose |
|---|---|---|
GPTBot | OpenAI | Collecting pages that may be used to train models |
OAI-SearchBot | OpenAI | ChatGPT's search feature. If you block this one, your site won't appear in ChatGPT's search answers |
ChatGPT-User | OpenAI | Fetching a page when a user asks for it. According to OpenAI, robots.txt may not apply here |
ClaudeBot | Anthropic | Collecting pages that may be used to train models |
Claude-SearchBot | Anthropic | Indexing pages for Claude's search feature |
Claude-User | Anthropic | Fetching a page when a user asks for it |
Google-Extended | Not a separate robot, but a name you use to indicate whether Google may use your content for training Gemini models and for grounding. According to Google, this doesn't affect your ranking in Google Search | |
PerplexityBot | Perplexity | Showing and linking sites in Perplexity's search results, not for training |
Perplexity-User | Perplexity | Fetching a page when a user asks for it. According to Perplexity, it usually ignores robots.txt |
The list changes regularly. For the current names, check the documentation from OpenAI, Anthropic, Google and Perplexity.
2. An example robots.txt
This example blocks model training but allows search robots and regular search engines:
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Disallow: /winkelmand/ Sitemap: https://example.com/sitemap.xml
- A robot only follows the group with its own name, and otherwise the
*group. So in the example, GPTBot isn't allowed anywhere, andDisallow: /winkelmand/doesn't apply to GPTBot. - If you want to allow all AI robots, you don't need to do anything: without rules, everything is allowed.
- If you accidentally block a search robot, such as
OAI-SearchBot,Claude-SearchBotorPerplexityBot, you'll be harder to find in those AI search services. - Blocking only works going forward. It won't retrieve anything that has already been fetched.
public_html folder. You create or edit it via File Manager or FTP. WordPress generates its own robots.txt as long as no file exists, and the AI Builder places one there automatically when you publish. To learn more about how robots.txt works, read Keeping a page out of Google: robots.txt or noindex?.3. What is llms.txt?
llms.txt is a 2024 proposal for a text file at https://example.com/llms.txt. In it, you write a short description of your site in Markdown along with links to the most important pages, so an AI service can quickly see where the right information is. A simple example:
# Example Bakery > Artisan bakery in Bristol offering bread, pastries and cakes to order. ## Important pages - [Products](https://example.com/assortiment): bread, pastries and cakes - [Ordering](https://example.com/bestellen): order and collect a cake - [Contact](https://example.com/contact): address and opening hours
Good to know:
- llms.txt is not an official standard, but a proposal. There's no guarantee that AI services will read the file. Google has stated that Google Search doesn't use it.
- llms.txt doesn't block anything. If you want to keep robots out, use robots.txt.
- The file isn't created automatically: you make it yourself, or have a plugin create it.
4. Creating llms.txt in WordPress
The well-known SEO plugins can create and maintain llms.txt for you:
- Yoast SEO: go to Yoast SEO, Settings, Site features and enable llms.txt under the AI features. You can let Yoast choose the pages or select them yourself.
- Rank Math: in the Rank Math Dashboard, enable the LLMS Txt module. You'll then find the settings under General Settings.
Menu names vary by version. If you don't see the option, update the plugin. Then view the file at https://yourdomain.co.uk/llms.txt.
Without WordPress, you create the file yourself: a text file llms.txt in the public_html folder, just like robots.txt.
Detailed explanation with screenshots: Sitemap and robots.txt for WordPress and your own site and Sitemap and robots.txt for the AI Builder in the Getting found online guide.
Want to learn more? Take the free Getting found online course, lesson 3.3 robots.txt: what you should and shouldn't block.