Free technical SEO tool
Control how search engines and AI systems access your website. Check an existing robots.txt file or generate one based on which search, AI retrieval and model-training crawlers you want to allow.
Each crawler follows the most specific group that names it. If none does, the User-agent: * group applies. A crawler with no rule at all is allowed by default, which is not the same as blocked.
Errors affect search crawling. Warnings need a decision. Notes are for information, including publisher preferences that are not SEO problems.
robots.txt controls crawling. It is not a reliable way to keep a URL out of search results. Google notes that a blocked URL can still appear in Search, without a snippet, if other pages link to it. For pages that must not be indexed, allow crawling and use a noindex meta tag or header, or protect them with authentication.
Load the detected settings into the generator, adjust the crawlers and paths, and produce an updated file.
A robots.txt file tells crawlers which URLs they may request on one host. It manages crawling. It is not access control and it is not a reliable index-removal tool.
The file lives at the root and applies only to that protocol, host and port. Rules are grouped by user agent, and a crawler follows the single most specific group that names it, or the * group if none does. Groups are not cumulative: a crawler with its own group ignores the general one.
Google says a robots.txt disallow is not a mechanism for keeping a page out of Search. A blocked URL can still be indexed from links on other pages, and shows without a description. Worse, a blocked page cannot be crawled, so a noindex tag on it is never seen.
The file is public, so listing /private/ advertises the path. Anyone, including crawlers that ignore robots.txt, can request it. Use authentication or server-side access rules for anything sensitive, and keep robots.txt for crawl management.
The important distinction is not search versus AI. It is what each crawler is for. The major providers now publish separate user agents for search discovery, user-requested retrieval and model training, and each is controlled separately.
Fetch pages for Google Search and Bing Search. Blocking them limits crawling for those search engines. Almost every public website should allow them.
Find and fetch pages so AI search features can surface and cite them, or so an assistant can read a page a user asked about. OpenAI recommends allowing OAI-SearchBot for ChatGPT search. Allowing these makes citation possible; it does not guarantee it.
Collect content that may be used to train models, or in Google’s case a token that opts crawled content out of Gemini training and grounding. Blocking these is a publisher preference. It does not block the search crawlers above.
There is no single correct robots.txt. Pick the situation closest to yours, read why it is set that way, and apply it in the generator with one click. Every choice stays editable afterwards.
Most public websites
A business site, blog or publication that wants search traffic and wants to be cited by ChatGPT, Claude and other AI answers, but would rather not feed model training.
Training tokens are separate from search crawlers, so blocking GPTBot, ClaudeBot and Google-Extended costs no visibility in Google, Bing, ChatGPT search or Claude. It is an opt-out, not a wall.
Open by design
Documentation, open-source projects, public-interest content or a site that simply wants maximum reach and has no objection to training use.
The cleanest file is two lines. Naming crawlers you allow adds nothing for them, so the generator leaves them out unless you set one explicitly.
Online shop
Product and category pages should be crawlable by everyone you allow. Cart, checkout and internal search results only waste crawl budget and can leak session URLs into indexes.
/cart/ /checkout/ /search/Path restrictions in the general rule apply to every crawler that has no group of its own. When you block or explicitly allow a crawler, the generator repeats those paths inside its group, because groups do not combine. Check the real paths on your platform first.
No AI at all
A site that, for legal, licensing or editorial reasons, wants no AI system to fetch its pages, and accepts that it will not appear in AI search answers.
This removes you from ChatGPT search, Claude search and similar features, not only from training. Some user-triggered fetchers ignore robots.txt, so a strict policy also needs server or CDN rules.
Not ready yet
A development copy, a client preview or an internal tool that should not be crawled by anyone.
One general rule does it. Remember two things: robots.txt is not access control, so protect anything sensitive with authentication; and this file must not ship to production, which is the single most damaging robots.txt mistake.
Selective
Perhaps you allow OpenAI training under a licensing deal but block others, or you allow Perplexity and Mistral search but not Meta. Custom covers it.
Set each row in the generator. Use Allow when you want the file to state the permission, Block for a Disallow group, and Not mentioned when the general rule should decide. Add less common crawlers from the list.
Most problems are structural rather than exotic. The checker reports each with the line number and what a crawler will actually do.
Disallow: / under User-agent: * is normal on staging and a disaster when it ships to production. The checker flags it as a warning and shows every search crawler as blocked.User-agent: Googlebot with Allow: / under a general group that disallows /admin/ lets Googlebot into /admin/. A specific group replaces the general one, so repeat the restrictions in it. The generator does this for you.noindex line in robots.txt has been ignored by Google since September 2019, and a crawl-blocked page never shows Google its meta noindex. Allow the crawl and add the tag to the page.User-agent belong to no group and are ignored.Sitemap: sitemap.xml is invalid; the URL must be absolute. Disallow: admin does not match /admin/ because matching starts from the beginning of the path.Crawl-delay is honoured by Bing and Anthropic and ignored by Google. It is not wrong, but it is not a universal control. The checker labels it rather than failing it.The tokens the tool checks and generates, with each provider’s own description of purpose. Crawler information last reviewed: .
| Provider | Crawler | Purpose | robots.txt token | Source |
|---|---|---|---|---|
| Google Search | Crawls pages for Google Search. | Googlebot | Documentation | |
| Microsoft | Bing Search | Bing’s standard crawler, which handles most of Bing’s crawling. | bingbot | Documentation |
| OpenAI | ChatGPT search | Used to surface websites in search results in ChatGPT’s search features. | OAI-SearchBot | Documentation |
| Anthropic | Claude search | Navigates the web to improve search result quality for Claude users. | Claude-SearchBot | Documentation |
| Anthropic | Claude user requests | Fetches a page when a Claude user asks a question that needs it. | Claude-User | Documentation |
| OpenAI | OpenAI model training | Crawls content that may be used to train OpenAI’s generative AI foundation models. | GPTBot | Documentation |
| Anthropic | Anthropic model development | Collects web content that could contribute to training Anthropic’s models. | ClaudeBot | Documentation |
| Google Gemini model use | A robots.txt token, not a separate crawler. It controls whether content Google crawls may be used to train future Gemini models and for grounding. | Google-Extended | Documentation |
| Provider | Crawler | Purpose | robots.txt token | Source |
|---|---|---|---|---|
| Apple | Apple search (Siri, Spotlight, Safari) | Apple’s crawler for the search features in Spotlight, Siri and Safari. | Applebot | Documentation |
| Apple | Apple model training | A token, not a crawler. Controls whether content crawled by Applebot may be used to train Apple’s generative foundation models. | Applebot-Extended | Documentation |
| Perplexity | Perplexity search | Surfaces and links websites in Perplexity search results. Perplexity states it is not used to crawl content for AI foundation models. | PerplexityBot | Documentation |
| DuckDuckGo | DuckDuckGo search | DuckDuckGo’s crawler for improving its search results. | DuckDuckBot | Documentation |
| DuckDuckGo | DuckDuckGo AI answers | Crawls pages in real time for DuckAssist AI-assisted answers, which cite their sources. Not used to train AI models. | DuckAssistBot | Documentation |
| Meta | Meta AI training | Crawls content for training Meta’s foundation AI models or improving products by indexing content directly. | meta-externalagent | Documentation |
| Amazon | Amazon (Alexa and AI models) | Crawls to improve Amazon products and services, including Alexa answers, and may be used to train Amazon AI models. | Amazonbot | Documentation |
| Common Crawl | Common Crawl dataset | Builds Common Crawl’s open repository of web crawl data, which is freely available to anyone and commonly used to train AI models. | CCBot | Documentation |
| Yandex | Yandex search | Yandex’s main indexing robot. | YandexBot | Documentation |
| Baidu | Baidu search | Baidu’s search crawler, which visits pages to build the Baidu index. | Baiduspider | Documentation |
| Mistral | Mistral search index | Automated crawling for Mistral search indexing only. Mistral states this content is not used for generative AI training. | MistralAI-Index | Documentation |
| Mistral | Mistral user requests | Fetches a page when a user asks Mistral’s assistant a question that needs it. Not used for automatic crawling. | MistralAI-User | Documentation |
| Mistral | Mistral model training | Crawls web content to build datasets for training Mistral’s generative AI models. | MistralAI-Training | Documentation |
Official documentation: Google · Microsoft · OpenAI · Anthropic · Apple · Perplexity · DuckDuckGo · Meta · Amazon · Common Crawl · Yandex · Baidu · Mistral. The checker reports the optional crawlers only when a file names them. Not included because their providers state that robots.txt rules may not apply: OpenAI’s ChatGPT-User, Perplexity’s Perplexity-User, Meta’s meta-externalfetcher. Not included because no official documentation could be verified: ByteDance’s Bytespider, Naver’s Yeti. Crawlers are added only when the token and purpose can be verified from current provider documentation.
A plain-text file at the root of a host, such as https://www.example.com/robots.txt, that tells crawlers which paths they may request. It is defined by RFC 9309 and read by search engines and most AI crawlers before they fetch anything else. It applies only to the exact protocol and host it sits on.
No. It stops the page from being crawled. Google can still index the URL from links on other pages and show it without a description. To keep a page out of results, let it be crawled and add a noindex meta tag or X-Robots-Tag header, return a 404 or 410, or put it behind authentication.
There is no universal answer, because the crawlers serve different purposes. Search and retrieval crawlers such as OAI-SearchBot, Claude-SearchBot and Claude-User make your pages discoverable and citable in AI search features. Training crawlers such as GPTBot and ClaudeBot, and the Google-Extended token, relate to model development. Many publishers allow the first group and decide the second on their own terms. The generator keeps the decisions separate so you are not forced to treat them as one.
Both belong to OpenAI. GPTBot crawls content that may be used to train OpenAI’s models; disallowing it signals that your content should not be used for training. OAI-SearchBot surfaces websites in ChatGPT’s search features, and OpenAI recommends allowing it if you want to appear there. OpenAI states that each setting is independent of the others.
A robots.txt token, not a separate crawler. Google crawls with its normal user agents and uses the Google-Extended token to decide whether that content may be used to train future Gemini models and for grounding. Blocking it is done with User-agent: Google-Extended and Disallow: /.
No. Google states that Google-Extended does not impact a site’s inclusion in Google Search and is not used as a ranking signal. Googlebot is controlled separately.
ClaudeBot collects web content that could contribute to training Anthropic’s models; blocking it signals that future material should be excluded from training datasets. Claude-SearchBot improves search result quality for Claude users; blocking it prevents Anthropic from indexing your content for search. Claude-User is a third agent that fetches pages when a user asks Claude to read them. Each is controlled by its own group.
No. The file is public and crawlers are free to ignore it. Listing a path tells everyone where it is. Protect private content with authentication or server-side access controls, and use robots.txt only to manage crawling of pages that are meant to be public.
At the root of the host, in lowercase, as UTF-8 plain text: https://www.example.com/robots.txt. It covers only that host and protocol. example.com, www.example.com and shop.example.com each need their own file, and an http version does not cover https.
It is a useful, well-supported convention. Add Sitemap: followed by the absolute URL, and repeat the line for each sitemap. It is not required, because sitemaps can also be submitted in Search Console and Bing Webmaster Tools, so the checker treats a missing line as a recommendation rather than an error.
Technical SEO
robots.txt is one part of a site’s technical search setup. Crawling, indexation, canonicals, sitemaps, redirects and site architecture need to work together, and the fix is rarely a single file.
Explore Technical SEOOr ask about your site