Tag
crawlers
2 posts
SEO
Do Robots.txt Blocks Prevent ChatGPT Scraping?
Yes for documented crawlers, no for the wider system: what each OpenAI bot feeds, what blocking cannot reach, and the commercial calculus by site type.
AI search
how do I know if chatgpt is pulling my content
Check your server logs for OpenAI's three user agents: GPTBot, OAI-SearchBot, and ChatGPT-User. Each one means something different about how ChatGPT uses you.