Allow the search and retrieval crawlers, and decide the training crawlers on principle. For most sites that means allowing OAI-SearchBot, PerplexityBot and Claude-SearchBot, which feed AI answers, while GPTBot and ClaudeBot, which collect training data, are a business choice. Blocking a training bot costs little visibility. Blocking a search bot can remove you from answers.
This holds for sites that want to be found in ChatGPT, Perplexity and Claude and have no paid or licensed content to protect. A publisher with a licensing deal may block more.
Robots.txt is a request, not a lock. It cannot verify who is asking, it cannot undo past collection, and the vendors say user-triggered bots may not follow it.
If you would rather have someone audit how AI crawlers and LLMs read your site, our LLM SEO work covers it. We check what each bot can reach and fix what blocks the pages you want cited.
The Three Jobs An AI Crawler Can Do
Most robots.txt advice on AI bots starts with a list of names. That is the wrong starting point. Names change, vendors add tokens, and a list of brands tells you nothing about what happens when you block one.
Start with the job. Every AI crawler does one of three things, and the setting you pick follows the job, not the logo.
- Training: The bot collects public web content that may be used to train a vendor's generative models. ClaudeBot and GPTBot do this.
- Search indexing: The bot builds the index that powers an AI search answer. Claude-SearchBot, OAI-SearchBot and PerplexityBot do this.
- User-triggered retrieval: The bot fetches a page because a person asked a question that needs it. Claude-User, ChatGPT-User and Perplexity-User do this.
The job decides the setting. Anthropic's support article names ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for user-triggered retrieval. OpenAI's bot documentation treats GPTBot as the training control and OAI-SearchBot as the control for ChatGPT search. Perplexity's bot guide says PerplexityBot is not used for training.
That split is the whole game. If you want to be quoted in AI answers but keep your pages out of training sets, you can have both. The vendors built separate tokens so you could.
Here is the part most site owners miss. Blocking a training bot is a statement about the future of a model. Blocking a search bot is a statement about whether you appear in an answer next week. Those are different decisions with different costs, and one robots.txt rule for "AI bots" mixes them up.
AI Crawler Comparison At A Glance
The default for a site that wants AI visibility is to allow the search and retrieval bots and make a deliberate choice on the training bots.
| Bot | Vendor | Job | What Blocking It Stops | Default For An AI-Visibility Site |
|---|---|---|---|---|
| ClaudeBot | Anthropic | Training | Use of your content in future Claude training datasets | Your choice |
| Claude-SearchBot | Anthropic | Search indexing | Indexing for Claude search, which may reduce visibility in search responses | Allow |
| Claude-User | Anthropic | User-triggered retrieval | Claude fetching your pages for a user query, which may reduce visibility in user-directed search | Allow |
| GPTBot | OpenAI | Training | Use of your content to train OpenAI foundation models | Your choice |
| OAI-SearchBot | OpenAI | Search indexing | Inclusion in ChatGPT search answers | Allow |
| ChatGPT-User | OpenAI | User-triggered retrieval | Little, because robots.txt rules may not apply to it | Allow |
| PerplexityBot | Perplexity | Search indexing | Being surfaced and linked in Perplexity results | Allow |
| Perplexity-User | Perplexity | User-triggered retrieval | Little, because it generally ignores robots.txt | Allow |
Two rows deserve a second look. The "Your choice" rows are the only ones where a reasonable site can go either way. Everything else, if you want to be found, stays open.
The ChatGPT-User and Perplexity-User rows say "little" for a reason. A robots.txt rule is the wrong tool for those two. The next sections explain why, one vendor at a time.
ClaudeBot, Claude-SearchBot And Claude-User
No, blocking ClaudeBot does not stop Claude from citing your site. Anthropic runs three separate tokens, and ClaudeBot is only the one that collects training data. Anthropic's support article says blocking ClaudeBot signals exclusion from future training datasets and does not by itself stop user-triggered retrieval, which is Claude-User. The exception is a site that also blocks Claude-SearchBot or Claude-User, because those two are what let Claude read and surface a page.
That distinction is the one most pages on ClaudeBot skip. They define the bot and stop. The useful question is what each of the three tokens controls.
- ClaudeBot: Collects public web content that may be used to train Anthropic's generative AI models. Blocking it signals that your content should be left out of future training datasets. It has no direct effect on whether Claude can retrieve a page for a user.
- Claude-SearchBot: Indexes content for search. Blocking it stops that indexing, which may reduce your visibility in search responses.
- Claude-User: Retrieves content when a user asks Claude something that needs a page. Blocking it stops Claude retrieving your content for user queries, which may reduce visibility in user-directed search.
So a site that wants to stay visible and opt out of training blocks one token and leaves two open. A site that blocks all three has opted out of Claude entirely, and should expect to disappear from its answers.
ClaudeBot also supports a Crawl-delay directive, which matters if the real problem is server load rather than principle. Slowing a crawler is a different choice from refusing it, and many sites that think they need to block only need to slow down.
You may see Claude-Web on third-party lists of Anthropic bots. It does not appear in Anthropic's current token list, so there is no reason to write a rule for it. Stick to the three tokens Anthropic documents.
Once the bots can read your pages, being cited is a content question. That is the work of Answer Engine Optimization, which turns pages a crawler can reach into pages an engine wants to quote.
If you block ClaudeBot, should you still allow Claude-SearchBot? Yes, if you want search visibility. The two tokens do different jobs, and Anthropic documents them as separate controls.
GPTBot Vs OAI-SearchBot Vs ChatGPT-User
GPTBot and OAI-SearchBot are separate crawlers with separate jobs. OpenAI's bot documentation describes GPTBot as the control for whether site content is used to train foundation models, and OAI-SearchBot as the control for whether a site appears in ChatGPT search. Blocking GPTBot does not hide your site from ChatGPT search. Blocking OAI-SearchBot does remove you from search answer inclusion.
The same page says changes to a robots.txt rule can take about 24 hours to apply for OAI-SearchBot. Plan for a day before you judge whether a change worked.
| Bot | Job | What Blocking It Does |
|---|---|---|
| GPTBot | Training | Tells OpenAI not to use your content to train foundation models |
| OAI-SearchBot | Search indexing | Removes your site from ChatGPT search answers |
| ChatGPT-User | User-triggered retrieval | Has uncertain effect, because robots.txt rules may not apply |
The mistake people make is blocking GPTBot because it sounds like "the ChatGPT bot", then wondering why they still appear in ChatGPT search. They appear because OAI-SearchBot is a different bot and was never blocked. The reverse mistake is worse. A site that blocks OAI-SearchBot to protect its content from training has not protected anything from training, and has removed itself from the answers.
ChatGPT-User is the third piece. It acts when a person asks ChatGPT to look at a page, so it is user-triggered, and OpenAI says robots.txt rules may not apply to it. Treat a rule aimed at it as a request that may not be honoured, and do not build a policy that depends on it.
If your goal is simple, the OpenAI side is simple. To stay in ChatGPT search and stay out of training, allow OAI-SearchBot and block GPTBot. To stay in both, allow both. To leave both, block both and accept that ChatGPT search will not include you.
PerplexityBot And Perplexity-User
PerplexityBot is the search crawler. Perplexity's bot documentation says it surfaces and links sites in Perplexity search results, that it is not used for training, and that Perplexity recommends allowing it. That makes it the easiest decision in this guide. There is no training trade-off to weigh, so blocking it only costs you appearances in Perplexity answers.
Perplexity-User works differently. It is triggered when a person asks Perplexity a question, and the same documentation says it generally ignores robots.txt. A rule aimed at Perplexity-User is therefore not a control. It is a statement of preference that the bot may not read.
This matters for anyone aiming at goal three, a full opt-out. You can write a Disallow line for Perplexity-User, and it costs nothing, but you should not tell your legal team or your publisher partners that it works. If a page must stay out of reach of user-triggered fetches, robots.txt cannot do that on its own, and the fix lives at the server or access layer.
For everyone else, the practical answer is short. Allow PerplexityBot. Leave Perplexity-User without a block, since a block would not be honoured, and move on to the next decision.
What Allowing AI Crawlers Costs And Returns
Allowing a crawler costs bandwidth and, for training bots, some control over how your content is reused. It returns visibility in AI answers. The honest way to weigh that is with numbers, and two are worth having.
The first is volume. Across a sample of 85 connected sites, Trakkr recorded 568,865 ClaudeBot requests between 18 July and 17 August 2026. That is a 30-day window on a small sample, so read it as a sense of scale, not a benchmark for your site. A single site in that sample sees thousands of ClaudeBot requests a month on average, not millions.
The second is what comes back. Ecorpit's analysis puts Anthropic's crawl-to-referral ratio at 1,917 crawls for every referral in July 2026, down from 3,386 to 1 in June 2026 and 38,744 to 1 in July 2025. The same source puts OpenAI at 251 to 1 in July 2026, down from 647 to 1 in June 2026 and 1,104 to 1 in July 2025.
| Vendor | July 2025 | June 2026 | July 2026 |
|---|---|---|---|
| Anthropic | 38,744 to 1 | 3,386 to 1 | 1,917 to 1 |
| OpenAI | 1,104 to 1 | 647 to 1 | 251 to 1 |
Read the table honestly. Both ratios are falling fast, which means crawlers are sending more traffic back per request than a year ago. They are also still lopsided. Anthropic sent one referral for every 1,917 crawls in July 2026, and OpenAI one for every 251.
What does that mean for the allow decision? Training crawlers take far more than they return, because training is not meant to send visitors. Search crawlers are where referrals and citations come from. So the cost of allowing a training bot is real and mostly about principle, while the cost of blocking a search bot is lost presence in an answer.
Most businesses should allow most AI crawlers, since blocking them hides a business from AI answers. Selective blocking makes sense for paid content or licensing deals. If you run a subscription publication, your calculation differs from a SaaS blog that wants every mention it can get.
One more cost sits outside robots.txt. Even when an engine cites you, many users read the answer and never click, which is why zero-click queries caused by AI answers deserve their own measurement. Being present in the answer is the prize, and the click is a bonus. Building for that outcome is what Generative Engine Optimization is about.
Which Setting To Choose For Your Goal
To stay visible and opt out of training, allow the search bots and block the training bots. Three goals cover almost every site, and each has a clean setting.
Work through the steps in order and stop at the first one that describes you.
- Decide whether you have content to protect. If you sell content behind a paywall or have a licensing deal, goal three is probably yours. If you want every mention you can get, goal one fits.
- Decide whether training use bothers you. If it does not, choose goal one. If it does but you still want to appear in AI answers, choose goal two.
- Check the conflict. Blocking Claude-User and Claude-SearchBot reduces visibility, so a goal-three site gives up Claude answers by design.
- Accept the limit. Even goal three cannot control user-triggered bots that ignore robots.txt, namely Perplexity-User and, possibly, ChatGPT-User.
| Bot | Goal 1 Visible, Training Fine | Goal 2 Visible, No Training | Goal 3 Opt Out Of All |
|---|---|---|---|
| ClaudeBot | Allow | Block | Block |
| Claude-SearchBot | Allow | Allow | Block |
| Claude-User | Allow | Allow | Block |
| GPTBot | Allow | Block | Block |
| OAI-SearchBot | Allow | Allow | Block |
| ChatGPT-User | Allow | Allow | Block, may be ignored |
| PerplexityBot | Allow | Allow | Block |
| Perplexity-User | Allow | Allow | Block, generally ignored |
Goal two is where most thoughtful sites land. It keeps you in ChatGPT search, Claude search and Perplexity, and it tells OpenAI and Anthropic you do not want your pages in their training sets. It costs very little visibility, because the bots you blocked were never the ones producing citations.
Goal three is the expensive one. It is honest for a publisher with something to protect, and a poor trade for a company that wants customers to find it through AI answers.
Once the rules are in place, the harder part is the ongoing work. That means tracking what the bots fetch and what the engines cite. Our GEO and SEO Growth Programme handles that month after month, so the setting you chose keeps paying off.
Copy-Ready Robots.txt Blocks For Each Goal
Each block goes in the top-level robots.txt file, at the root of your domain. Every token needs its own User-agent line, and the syntax is the one the vendors document, a User-agent line naming the token followed by a Disallow or Allow rule. Replace any older AI bot rules you already have rather than stacking new ones on top, since a duplicate group for the same token is a common source of confusion.
Goal one, visible and training is fine. This block allows every documented token.
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
It leaves everything open. If you already allow all crawlers by default, this block changes nothing and only makes your intent explicit.
Goal two, visible and no training. This block blocks the two training bots and leaves the search and retrieval bots open.
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
It leaves search indexing and user-triggered retrieval open, so ChatGPT search, Claude search and Perplexity can still surface your pages.
Goal three, opt out of everything. This block blocks all eight tokens.
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
It leaves one gap open. User-triggered bots may ignore these lines, so a determined user-initiated fetch can still reach a public page.
If you only want to slow ClaudeBot rather than refuse it, a Crawl-delay line in its group is the documented route, since Anthropic says ClaudeBot supports that directive.
After you publish the file, run a full site crawl to check your rules against real URLs and see what your own site returns for each path. A crawler such as Semrush is one way to do that across the whole site in one pass, and it is worth doing before you trust a change to a file that applies to every page you own.
How To Check The Rules Work In Your Server Logs
A rule you cannot observe is a rule you are guessing about. The logs tell you what happened, and the check takes under an hour.
- Confirm robots.txt returns a 200 status. Request
/robots.txton your domain and check the status code in your access log. The expected result is a 200 response and the file you just published. A 404 or a redirect loop means the bots never saw your rules. - Filter the log by each token. Search for the user-agent string, for example
grep -i "ClaudeBot" access.log, and repeat for each token you wrote a rule for. Note the request count for each over the previous seven days. - Record the baseline. Count requests to content pages, not to robots.txt or images, for each bot before the change. This is the number you compare against.
- Wait out the change window. OpenAI says changes can take about 24 hours to apply to OAI-SearchBot, so give every bot at least a day, and ideally a week, before you read the results.
- Compare after the change. A blocked bot should stop requesting content pages, and keep requesting robots.txt. An allowed bot should keep requesting content pages at roughly its old rate.
- Check that the sender is real. User-agent strings can be spoofed, so a log entry naming a bot does not prove the request came from that vendor. Look for the verification details each vendor publishes on its own bot page before you rely on an IP range. This guide does not assert a verification method for any vendor.
- Cross-check with analytics. If you prefer reports to raw logs, content audit reports from GA4 and Search Console can show whether traffic from AI sources moves after the change.
- Make it a recurring task. A crawler audit is worth repeating quarterly, and AI SEO tools and agile sprints shows how to turn a check like this into a sprint item.
What should you do if a blocked bot keeps requesting content? First, check the sender. A spoofed user-agent is more likely than a bot ignoring its own vendor's documented rule. Second, check the rule syntax and that the file is at the top level of the domain, because a misplaced file would hide your rules. Only then should you suspect the bot.
What Robots.txt Cannot Do
Robots.txt sets policy. It does not enforce it. Four limits matter.
- It cannot verify the sender. Robots.txt sets an access policy for ClaudeBot but cannot verify who is asking, so a request that claims to be ClaudeBot may be anyone.
- It cannot undo past collection. A block applies from the time the bot next reads the file, so content already collected stays collected.
- Blocking by IP can backfire. Anthropic says blocking by IP address is not a reliable opt-out, because it can stop the bot from reading robots.txt at all, and a bot that cannot read your rules cannot follow them.
- User-triggered bots may ignore it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, so a rule aimed at either is a preference and not a control.
Changes are also not instant. Allow a day for OAI-SearchBot and longer for any vendor that does not document its refresh time.
Fixing access is the first half of AI visibility. The second half is giving the bots something worth citing. Once crawlers can reach the site, stale pages lose AI citations, so content refresh and decay prevention is the next job. If your older posts are the ones being fetched, rewriting old posts for AI search visibility covers the tools. And if you need to decide which pages are worth exposing in the first place, AI-ready content auditing software is the route to look at.
FAQs
Does Blocking ClaudeBot Remove My Site From Claude Answers?
Is GPTBot The Same As OAI-SearchBot?
Will Blocking GPTBot Hide My Site From ChatGPT Search?
Do User-Triggered Bots Such As ChatGPT-User And Perplexity-User Follow Robots.txt?
How Long Does A Robots.txt Change Take To Apply To OAI-SearchBot?
Does ClaudeBot Support Crawl-Delay?
Can I Block AI Crawlers By IP Address Instead?
Does Blocking A Crawler Remove Content It Already Collected?
Should I Allow Claude-SearchBot If I Block ClaudeBot?
If your logs show a bot this guide does not cover, or you are unsure which setting fits your content, send us the question. One of the founders reads every message and replies with what we would do first.
📑 Disclosure!
Some links in this article are affiliate links. If you sign up through them, The Rank Masters may earn a commission at no additional cost to you. This does not influence which tools we recommend or what we say about them.
