SQSEO-Bot

A polite crawler
that reads the pages AI answers cite.

When ChatGPT, Perplexity or Google cite a page in an answer we measured, SQSEO-Bot reads that page once so our customers can see what it says. This page explains exactly what it does and how to stop it.

What it does

One shared read per page,
never a crawl of your site.

Only pages AI answers cite
SQSEO-Bot never discovers pages by following links. It fetches a URL only after an AI answer we observed retrieved or cited it, and it keeps one snapshot of the readable text shared by every customer, never one copy per customer.
robots.txt first, every time
Before any fetch it reads your robots.txt and honours Disallow rules for sqseo-bot (or *) and your Crawl-delay. If robots.txt cannot be read, the whole host is skipped.
Rarely, and gently
At most one request per host every 10 seconds, a page revisited at most weekly, with If-None-Match and If-Modified-Since so an unchanged page costs you a 304 and nothing more. No login, no JavaScript, no CAPTCHA solving, ever.
Identifies itself honestly
Every request carries the User-Agent below and comes from our own servers. We never imitate a browser, rotate identities or use proxies.
The User-Agent on every request
SQSEO-Bot/1.0 (+https://sqseo.com/bot)
Blocking it

Two lines in robots.txt
and we stop within a day.

Add this to your robots.txt. We re-read robots.txt at least once every 24 hours, a blocked page is never fetched again, and stored snapshots of your pages expire 90 days after the fetch.

User-agent: sqseo-bot
Disallow: /

Prefer to ask directly? Send us the domain through the contact form

What we keep

The readable text,
for 90 days.

A snapshot is the page's visible text as Markdown: headings, paragraphs, lists, links. No scripts, no styles, no cookies, no form data, no images. It exists so a customer can see why an AI answer cited a page, and it expires 90 days after the fetch.