← Knowledge
Tag

robots.txt

4 posts

A paywalled article served to logged-in users but showing only a login wall to AI answer-engine crawlers
AI search

how do i prevent ai answer engines from summarizing gated content

Two levels: ask crawlers not to via robots.txt, and enforce it by never serving gated content to unauthenticated bots. Only the second is real protection.

·10 min
How to block ClaudeBot for AI training while keeping Claude-SearchBot and Claude-User allowed in robots.txt for Claude visibility
AI search

How to block Claudebot but still show in Claude recommendations

You can opt out of AI training and still appear in Claude. Anthropic runs three separate crawlers, so block ClaudeBot in robots.txt and keep Claude-SearchBot and Claude-User allowed.

·10 min
Whether blocking Google-Extended lowers organic rankings, and how it differs from Googlebot
Keyword research

Does blocking Google-Extended lower your organic rankings?

Worried that blocking Google-Extended will tank your SEO? Google says it will not. Here is exactly what Google-Extended controls, what blocking it costs, and how to decide.

·10 min
How Perplexity AI crawls websites: PerplexityBot versus Perplexity-User and what robots.txt controls
SEO

Can Perplexity AI actually crawl your website?

Can Perplexity AI crawl your website, and can you stop it? Here is how PerplexityBot and Perplexity-User work, what robots.txt does and does not control, and what it means for visibility.

·10 min