robots-txt
6 posts
Will AI Indexing Become Monetized Directly?
Pay per crawl, RSL licensing, publisher revenue shares and ads in AI answers already exist. What is paid now, what is unlikely, and whether your site should charge.
Do Robots.txt Blocks Prevent ChatGPT Scraping?
Yes for documented crawlers, no for the wider system: what each OpenAI bot feeds, what blocking cannot reach, and the commercial calculus by site type.
how do i prevent ai answer engines from summarizing gated content
Two levels: ask crawlers not to via robots.txt, and enforce it by never serving gated content to unauthenticated bots. Only the second is real protection.
How to block Claudebot but still show in Claude recommendations
You can opt out of AI training and still appear in Claude. Anthropic runs three separate crawlers, so block ClaudeBot in robots.txt and keep Claude-SearchBot and Claude-User allowed.
Does blocking Google-Extended lower your organic rankings?
Worried that blocking Google-Extended will tank your SEO? Google says it will not. Here is exactly what Google-Extended controls, what blocking it costs, and how to decide.
Can Perplexity AI actually crawl your website?
Can Perplexity AI crawl your website, and can you stop it? Here is how PerplexityBot and Perplexity-User work, what robots.txt does and does not control, and what it means for visibility.