Plenty of site owners want the same thing: keep their content out of AI training data, but stay visible when Claude answers a user and recommends sources. Many assume that is impossible, that blocking Claude means blocking it entirely. It is not. Anthropic runs three separate crawlers with separate jobs and separate controls, so you can block the training one and keep the ones that make you visible in Claude. Here is exactly how, straight from Anthropic’s own documentation.
The short answer
To block ClaudeBot but still appear in Claude’s answers and recommendations, disallow ClaudeBot in robots.txt while leaving Claude-SearchBot and Claude-User allowed. ClaudeBot is Anthropic’s training crawler, Claude-SearchBot improves Claude’s search results, and Claude-User fetches a page when a user asks Claude about it. Anthropic states that blocking one does not affect the others, since each operates independently and is controlled by its own user-agent in robots.txt. So opting out of training while staying visible in Claude is a matter of disallowing the right bot and not the others. It is a precise, supported configuration, not a hack.
Anthropic runs three separate crawlers
The whole solution rests on one fact: Claude does not use a single bot. Anthropic’s documentation describes three distinct crawlers, each with its own user-agent and purpose.
| Crawler | User-agent | What it does | Block it to |
|---|---|---|---|
| ClaudeBot | ClaudeBot | Collects web content that may contribute to training | Opt out of AI model training |
| Claude-SearchBot | Claude-SearchBot | Crawls to improve Claude’s search result quality | Remove yourself from Claude search (not recommended if you want visibility) |
| Claude-User | Claude-User | Fetches a page when a user asks Claude about it | Stop user-initiated visits to your pages |
Per Anthropic, ClaudeBot collects web content that could contribute to training, Claude-User accesses websites when individuals ask Claude questions, and Claude-SearchBot navigates the web to improve search result quality. Three jobs, three controls.
Why blocking is independent
This is the load-bearing detail, so it is worth stating plainly. Anthropic confirms that blocking ClaudeBot does not automatically affect Claude-User or Claude-SearchBot, because each crawler operates independently and must be disabled separately in robots.txt. That independence is exactly what makes the goal achievable: you are not choosing between all-in and all-out, you are choosing per bot. Disallow the training crawler, leave the search and user crawlers allowed, and you have opted out of training while staying eligible to appear in Claude’s answers. If the bots were a single agent, this would be impossible; because they are three, it is straightforward.
How to block ClaudeBot but stay in Claude
The configuration is a few lines of robots.txt. Disallow ClaudeBot, and simply do not disallow the other two, since anything not disallowed is allowed by default. Being explicit, though, makes intent clear:
# Opt out of AI training, stay visible in Claude search and answers
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Anthropic states that all three bots honor robots.txt directives, including Disallow and the non-standard crawl-delay extension. So this configuration is respected: ClaudeBot is told not to crawl for training, while Claude-SearchBot and Claude-User remain free to crawl and fetch, which is what keeps you visible in Claude. The same robots.txt-first approach that governs crawl access generally is covered in can Perplexity AI crawl your website.
What each bot means for your visibility
It helps to map each bot to an outcome. Blocking ClaudeBot affects whether your future content can be used to train Anthropic’s models; it does not, by itself, remove you from Claude’s answers. Claude-SearchBot is the one tied to search visibility: Anthropic notes that blocking it means your content will not be indexed for Claude’s search, which can reduce your presence and accuracy in Claude-powered answers, so if you want visibility, leave it allowed. Claude-User is the live fetch when someone asks Claude about a specific page; allowing it lets Claude pull your page in context. So for the stated goal, block ClaudeBot, keep Claude-SearchBot and Claude-User. Tracking how you actually appear in Claude is covered in analyze Claude visibility.
The same pattern at OpenAI
This training-versus-search split is not unique to Anthropic, which is reassuring because it means the mental model transfers. OpenAI documents the same separation: GPTBot crawls content for training generative models, while OAI-SearchBot surfaces websites in ChatGPT’s search features, with ChatGPT-User handling user-initiated visits. So the same move applies there: block GPTBot to opt out of training, keep OAI-SearchBot allowed to stay in ChatGPT search. The cross-engine differences in how the search bots then rank you are in ChatGPT vs Perplexity ranking differences. The pattern is becoming an industry norm: separate training crawlers from search crawlers, and control them separately.
Why you might want training-out but search-in
The configuration only makes sense if the goal does, so it is worth naming the motivations. Some organizations have legal, licensing, or policy reasons to keep their content out of model training, while still wanting the referral traffic and authority that come from being cited in AI answers. Others simply prefer not to contribute to training without compensation but see clear value in AI-search visibility. In all these cases, the training-out, search-in posture is the rational middle ground, and Anthropic’s three-bot design supports it directly. You are not forced to choose between protecting your content from training and being discoverable in Claude.
What this does and does not do
Setting expectations avoids disappointment. Blocking ClaudeBot tells Anthropic to exclude your future content from training datasets; it is forward-looking and does not retroactively remove what was already collected. Leaving Claude-SearchBot and Claude-User allowed keeps you eligible to appear in Claude, but eligibility is not a guarantee of citation, since being cited still depends on authority, relevance, and clear answers. And robots.txt is a directive that compliant bots honor, not a hard access control; Anthropic states its bots respect it, but it is not a security boundary. So this is the right tool for the stated intent, with realistic limits.
Being allowed is not the same as being cited
Keeping Claude-SearchBot and Claude-User allowed makes you eligible to appear, but eligibility is only the entry ticket, and it is worth being honest about that. What turns eligibility into actual citations is the same thing across every AI engine: authority, relevance, and clear answers. Ahrefs, studying 75,000 brands, found that the factors most correlated with AI brand visibility align closely with established authority and relevance signals. Citations also reach well beyond the top of the rankings, with Ahrefs finding that only 38 percent of AI Overview citations come from top-10 pages, down from about 76 percent a year earlier, and they follow source preferences, with Profound finding that ChatGPT cites Wikipedia in about 47.9 percent of citations while Perplexity leans on Reddit at roughly 46.7 percent. So allowing the right crawlers is necessary but not sufficient: once you are eligible, being genuinely authoritative and clearly answering the question is what actually earns the mention.
How to verify it worked
Do not assume, confirm. After updating robots.txt, check that the file is reachable at your domain root and returns the expected rules, since bots must be able to read it for any of this to apply, which is why IP blocking alone is unreliable, it can stop a bot from reading robots.txt at all. Then monitor your server logs for the three user-agents to confirm ClaudeBot stops crawling while Claude-SearchBot and Claude-User continue. Over time, watch whether you still appear in Claude’s answers for your topics. Verification closes the loop: a rule you never confirmed is a rule you cannot trust.
A worked example
A publisher wanted out of AI training but relied on referral visibility, and had blocked Claude entirely with a blanket rule, losing its presence in Claude answers as a result. The fix matched the three-bot model: it replaced the blanket block with a ClaudeBot-only Disallow and left Claude-SearchBot and Claude-User allowed. Within its log data, ClaudeBot stopped crawling while the search and user agents resumed, and over the following weeks its content reappeared in Claude answers for its core topics. Nothing about its content changed; only the precision of its robots.txt did. The all-or-nothing assumption had cost it visibility it could have kept.
Common mistakes
A few errors undo the intent. The first is a blanket block of all Claude bots when you only meant to opt out of training, which also removes your search visibility. The second is blocking Claude-SearchBot by mistake, the very bot tied to appearing in Claude search. The third is relying on IP blocking, which can prevent bots from reading robots.txt and thus from honoring your opt-out. The fourth is assuming the change is retroactive, when it governs future crawling. The fifth is never verifying, so a typo in a user-agent silently breaks the setup. Precision and verification are the whole game here.
Keep the configuration under review
AI crawlers are a moving target, so a robots.txt you set once and forget can quietly drift out of date. Providers add, rename, or re-scope bots over time, and the line between training and search crawlers is exactly the kind of thing that gets refined. The low-effort habit is to re-check periodically: confirm the current user-agent names against each provider’s own documentation, make sure your ClaudeBot disallow is still in place, and verify the search and user agents are still allowed. Treat the provider docs as the source of truth rather than a third-party list that may lag. A few minutes a quarter keeps your training-out, search-in posture accurate as the crawlers evolve, which is far cheaper than discovering months later that a renamed bot slipped past your rules.
The bottom line
You can block ClaudeBot and still show up in Claude because Anthropic runs three independent crawlers: ClaudeBot for training, Claude-SearchBot for search quality, and Claude-User for live fetches, each controlled separately in robots.txt. Disallow ClaudeBot, leave Claude-SearchBot and Claude-User allowed, confirm robots.txt is readable, and verify in your logs. The same training-versus-search split exists at OpenAI, so the model transfers. Just remember that staying eligible is not the same as being cited, that still takes authority and clear answers, which is where question-level research from SQSEO and good content do the real work.
Frequently asked questions
Can I block ClaudeBot and still appear in Claude’s answers?
Yes. Anthropic runs three separate crawlers, and blocking one does not affect the others. ClaudeBot is the training crawler, while Claude-SearchBot improves Claude’s search results and Claude-User fetches pages when users ask Claude about them. Disallow ClaudeBot in robots.txt and leave Claude-SearchBot and Claude-User allowed, and you opt out of training while staying eligible to appear in Claude’s answers and recommendations.
What is the difference between ClaudeBot, Claude-SearchBot, and Claude-User?
Per Anthropic, ClaudeBot collects web content that may contribute to training Anthropic’s models, Claude-SearchBot navigates the web to improve the quality of Claude’s search results, and Claude-User accesses a website when an individual asks Claude a question that requires it. Each has its own robots.txt user-agent and is controlled independently, so you can allow or block them separately depending on your goal.
How do I write the robots.txt rules?
Add a block disallowing ClaudeBot, and do not disallow the other two, since anything not disallowed is allowed by default. For example: User-agent: ClaudeBot followed by Disallow: / opts out of training, while Claude-SearchBot and Claude-User remain allowed and keep you visible in Claude. Anthropic states all three bots honor robots.txt, so the rules are respected as long as your robots.txt file is reachable.
Does blocking ClaudeBot remove my content from Claude immediately?
No. Blocking ClaudeBot is forward-looking: it tells Anthropic to exclude your future content from training datasets, and it does not retroactively remove content already collected. It also does not, by itself, remove you from Claude’s search answers, because those depend on Claude-SearchBot and Claude-User, which you leave allowed. Robots.txt is a directive compliant bots honor, not a security control, so treat it as an opt-out signal rather than a hard barrier.
