How to create a robots.txt file
- Start from a preset, or edit the group for
*(every crawler without its own group). - Add Disallow rules for paths crawlers shouldn't fetch, and Allow rules for exceptions inside them.
- Add groups for specific crawlers, such as the AI crawlers below, and your sitemap URLs.
- Test real paths with the tester: it says whether a crawler may fetch each one, and which line decided.
- Download the file and upload it to the root of your site, so it's at
https://yoursite.com/robots.txt.
How Google matches rules
A crawler follows only one group: the one naming it most specifically, or * if none does. Groups for the same crawler are merged. Within the group, the rule with the longest path that matches the URL wins, and when an Allow and a Disallow are the same length, Allow wins. Paths are case-sensitive; user-agent names aren't.
| Rules | Path | Result |
|---|---|---|
Allow: /p, Disallow: / | /page | Allowed: /p is longer |
Allow: /folder, Disallow: /folder | /folder/page | Allowed: a tie goes to Allow |
Allow: /page, Disallow: /*.htm | /page.htm | Blocked: /*.htm is longer |
Allow: /$, Disallow: / | / | Allowed: $ matches the root only |
Allow: /$, Disallow: / | /page.htm | Blocked |
Disallow: /*.php$ | /index.php?x=1 | Allowed: $ means the URL ends there |
* matches any run of characters and $ marks the end of the URL. These examples come from Google's documentation, and the tester above gives the same answers.
AI crawler user agents
These are the robots.txt tokens each company documents. Training crawlers collect content that may be used to train AI models; AI search crawlers find pages to show and link in AI search answers, so blocking them can keep your site out of those answers. Robots.txt is a request these companies say they honour, not a technical block.
| Token | Company | Used for | Documentation |
|---|---|---|---|
GPTBot | OpenAI | AI training. Crawls content that may be used to train OpenAI's models. | Overview of OpenAI crawlers |
ClaudeBot | Anthropic | AI training. Collects content that may contribute to training Claude. | Does Anthropic crawl data from the web, and how can site owners block the crawler? |
Google-Extended | AI training. Controls use for Gemini training and grounding; not a separate crawler and no effect on Google Search. | Google's common crawlers (Google-Extended) | |
Applebot-Extended | Apple | AI training. Opts out of training Apple's foundation models; doesn't crawl or affect search. | About Applebot |
CCBot | Common Crawl | AI training. Builds Common Crawl's free web archive, which anyone can download, AI developers included. | CCBot |
meta-externalagent | Meta | AI training. Crawls for training Meta's AI models and indexing content. | Meta Web Crawlers |
OAI-SearchBot | OpenAI | AI search. Finds and links websites in ChatGPT search. | Overview of OpenAI crawlers |
Claude-SearchBot | Anthropic | AI search. Improves search results in Claude. | Does Anthropic crawl data from the web, and how can site owners block the crawler? |
PerplexityBot | Perplexity | AI search. Surfaces and links websites in Perplexity answers; not used to train foundation models. | Perplexity Crawlers |
Each link was checked on 7 October 2026. Companies add and rename crawlers, so check their pages before relying on this list. Blocking Google-Extended doesn't affect how your site appears in Google Search.
What robots.txt can't do
- Hide a page from Google. A blocked URL can still be indexed if other sites link to it. Use a noindex robots meta tag, and leave the page crawlable so Google can see it.
- Protect private content. The file is public and obeying it is voluntary. Use a password for anything private.
- Slow Googlebot down. Google ignores crawl-delay. Some other crawlers, such as Bingbot, read it.
- Cover other hosts. A file applies only to its own protocol, host and port:
www.example.comandexample.comeach need one.
Sources
Each link was last checked on .
- Introduction to robots.txt, Google Search Central: robots.txt manages crawling, isn't a way to keep a page out of Google, and is up to each crawler to obey.
- How Google interprets the robots.txt specification, Google Search Central: the file's location and scope, group selection by user agent, * and $ wildcards, longest-match precedence with allow winning ties, the 500 KiB limit, and that crawl-delay isn't supported.
- RFC 9309: Robots Exclusion Protocol, IETF (2022): the robots.txt syntax and matching rules the generator and path tester follow.
- Robots meta tag, data-nosnippet, and X-Robots-Tag specifications, Google Search Central: the index, noindex, follow and nofollow values, and that a page blocked by robots.txt can't show its noindex.
- Overview of OpenAI crawlers, OpenAI: GPTBot crawls for model training and OAI-SearchBot for ChatGPT search; both follow robots.txt.
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic: ClaudeBot collects content that may be used for training, and Claude-SearchBot improves search.
- Google's common crawlers (Google-Extended), Google Search Central: Google-Extended controls use for Gemini training and grounding, and doesn't affect inclusion or ranking in Google Search.
- About Applebot, Apple Support: Applebot-Extended opts content out of training Apple's foundation models without removing it from search.
- CCBot, Common Crawl: the CCBot token, and that CCBot follows robots.txt.
- Meta Web Crawlers, Meta for Developers: meta-externalagent crawls for AI training and follows robots.txt.
- Perplexity Crawlers, Perplexity: PerplexityBot indexes pages for Perplexity's search answers, isn't used for training foundation models, and follows robots.txt.
Common uses
- Opting out of AI training. Use the AI preset to add GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and Meta's crawler with Disallow: /, while search engines keep crawling as before.
- Keeping crawlers out of admin and cart pages. Disallow folders like /admin/ or /cart/ for all crawlers, then test a few real URLs to be sure product pages stay allowed.
- Checking a rule before it goes live. Type a path and a crawler and the tester says whether it's allowed and which line decided, so a stray Disallow: / never blocks the whole site.