Most AI visibility problems are not ranking problems, they are retrieval problems: the engine never had your page. Before optimising anything, find out which of your URLs an AI engine can actually fetch, parse and use, because a page that fails any of those three steps cannot be cited no matter how well it is written. SQSEO exists for the demand side of this, showing which questions matter, and the supply side audit below tells you whether you have anything retrievable to answer them with.
Three gates, not one
A page has to pass three independent checks before it can appear in an answer, and teams routinely optimise the third while failing the first.
Fetchability: can the crawler request the URL and get a 200 with content. Parseability: does the raw response contain your text, or does the text only appear after JavaScript runs. Usability: once chunked, does any passage make a complete, attributable claim.
Each gate has its own failure mode and its own test. Running all three across your key pages takes an afternoon and usually explains more than a month of content work.
Gate one: can the crawler get it
AI engines use their own crawlers, and their behaviour differs from Googlebot. The obvious starting point is your robots.txt, because several AI crawlers are commonly blocked by default in security templates or by a CDN rule nobody remembers adding.
List the user agents you intend to allow and check each explicitly rather than assuming a wildcard covers them. Google documents its own crawler fleet in the overview of Google crawlers, and the AI oriented agents from other providers each publish their own identifiers. A blanket disallow you inherited is the single cheapest visibility bug to fix.
Then check what a crawler actually receives. Bot protection, rate limiting and geographic rules frequently serve a challenge page or a 403 to anything that is not a consumer browser. The test is to fetch your own URL with a plain client and a bot user agent and read the status code. A 200 in your browser and a 403 to a bot is invisible in every analytics tool you own.
Gate two: is your text in the response
This is the gate that catches the most sites, and it takes one command to check: fetch the raw HTML and search it for a distinctive sentence from the middle of your article.
If the sentence is absent, your content is rendered client side and reaches only clients that execute JavaScript. Google documents the implications in its JavaScript SEO basics, and the situation is stricter for AI crawlers, several of which do far less rendering than Googlebot does.
The same test catches three related problems. Content behind a cookie wall or consent gate that blocks rendering. Content loaded on scroll or behind a tab or accordion that never expands for a crawler. And content inside an iframe, which is a separate document as far as extraction is concerned.
| Symptom | Likely gate | One command check | Typical fix |
|---|---|---|---|
| No AI citations at all, any query | Gate one | curl with a bot user agent, read status | Fix robots.txt or CDN rule |
| Cited only for homepage, never deep pages | Gate one | Check sitemap coverage and internal links | Improve discovery paths |
| Page loads fine, never cited | Gate two | curl and grep for a body sentence | Server render the content |
| Cited but answer is vague | Gate three | Read any 250 word window alone | Front load claims per section |
| Competitor content quoted from your page | Gate three | Check what boilerplate survives | Clean up template noise |
Gate three: is any passage usable
A page can be fetched and parsed and still contribute nothing, because retrieval works on passages. The test is manual and quick: take any 250 word window from the page, read it with no knowledge of the rest, and ask whether it makes a complete claim you could quote.
Windows that fail usually fail for one of three reasons. They open with a pronoun or a backreference, so the subject is unknown. They build toward a point that lands outside the window. Or they contain mostly navigation and boilerplate that survived extraction.
The fix is structural rather than additive: name the subject in each section, put the claim in the first sentence, and keep template furniture out of the article body. That is the same passage level reasoning covered in how much of your page an AI engine actually reads.
Discovery: the pages nothing links to
Fetchable and parseable pages still need to be found. Orphan pages with no internal links and no sitemap entry are functionally invisible, and large sites accumulate them steadily.
Pull your sitemap, pull a crawl of your own internal links, and compare. URLs in neither set are orphans. URLs in the sitemap but with no internal links are weakly discovered, which is not fatal but is a disadvantage.
Pay particular attention to pages created outside the main CMS flow: landing pages from campaigns, documentation on a subdomain, and anything published by a tool that does not write to the sitemap. These are frequently the most useful pages you own and the least discoverable.
Check what is actually being cited, not just what is retrievable
The audit above tells you what could be used. Running prompts tells you what is used, and the gap between those two lists is the actionable part.
Record the cited URLs for every run, not only whether your brand appeared. Three patterns show up repeatedly. Your brand mentioned with no citation means the model is drawing on background knowledge rather than your content, so publishing something retrievable is the move. A single old URL cited for everything means one page is doing all the work and the rest of the site is failing a gate, and the run counts you need before trusting any of this are covered in how stable are AI answers over time. And competitor URLs cited for questions about you means their content occupies the pool where yours should be, which is the situation behind competitors being cited instead of you.
Rendering budget and page weight
Even where a crawler renders, it does so within limits. Heavy pages with large JavaScript bundles, many third party scripts and slow server responses get partially processed or abandoned. Research on generative engine optimization focuses on content properties rather than delivery, but the delivery layer is a precondition: content that never arrives cannot have properties.
Practical checks are ordinary performance work with a different motivation. Time to first byte, total page weight, and the number of blocking requests before main content appears. A page that takes eight seconds to become readable is not merely a poor experience, it is a page some crawlers will give up on.
What about llms.txt and AI specific files
There is enthusiasm for AI specific manifest files, and they cost almost nothing to publish. Treat them as optional extras rather than as a fix for a failing gate, because a file describing your content does not help if the content itself is unfetchable or unparseable.
The order of operations matters. Fix robots and bot rules, server render your text, structure passages so they stand alone, ensure discovery, and only then add manifest files. Doing it in the reverse order is common and produces no measurable change, which then gets misread as the technique not working.
Why passage usability is a retrieval question, not a style question
It is tempting to treat gate three as editing advice. It is not: it is the mechanism retrieval runs on. Systems compare an encoded query against encoded passages, an approach established by dense passage retrieval and still the backbone of production search over documents. A passage with no clear subject and no complete claim encodes to a vague vector that matches nothing sharply.
That has a measurable consequence: two pages with identical facts can have completely different citation rates purely because one states its claims inside self contained windows and the other spreads them across paragraph boundaries. Nothing about authority, domain or backlinks explains the difference.
It also explains an odd observation teams make, where a short FAQ page outperforms a long definitive guide on the same topic. The FAQ is accidentally well formed for retrieval: each question is a subject, each answer is a complete claim, and each pair fits inside a chunk. The guide may be better writing and worse retrieval material.
Verify what the engine did with your page
Being cited is not the same as being represented correctly, and the audit should include a correctness pass. Work evaluating verifiability in generative search engines found that a meaningful share of generated statements were not fully supported by the sources cited alongside them, with precision varying widely between engines.
In practice that means checking, for each run where you are cited, whether the claim attributed to your page is actually on your page. Two failure modes recur. The model merges your passage with a neighbouring source and attributes the blend to you. Or it extracts a qualified statement without its qualifier, turning a conditional claim into an absolute one.
Both are partly addressable from the content side. Qualifiers that live in the same sentence as the claim survive extraction; qualifiers in an adjacent sentence frequently do not. Writing “for teams under fifty people, X is usually faster” is safer than writing the claim and then the constraint separately.
Turning the audit into a repeatable check
An afternoon audit that never runs again decays immediately, because sites change. The parts worth automating are the cheap ones.
A scheduled job that fetches a sample of URLs with a bot user agent and asserts a 200 plus the presence of a known body sentence catches both gate one and gate two regressions, and it catches them the week a CDN rule or a framework upgrade breaks rendering rather than a quarter later. Add your sitemap count as a tracked number, since a sudden drop is usually a build problem.
Gate three does not automate well, but it does not need to run often. Review it when you change templates, since template changes are what introduce boilerplate into chunks, and when you publish a new content type.
The point of making it repeatable is that these failures are silent. Nothing in analytics reports that a crawler stopped getting your text, and the first visible symptom is a slow decline in citations that looks like a content problem and is not.
A worked example: an audit that explained a year
A company with 180 published articles and near zero AI citations ran the three gates in an afternoon.
Gate one passed: robots.txt was permissive and bot fetches returned 200. Gate two failed comprehensively. The site was a client rendered single page application, and curl returned a shell containing a loading spinner and no article text on every URL. Every article had been invisible to every non rendering crawler since launch.
Gate three was untestable until gate two was fixed. After moving to server rendered output, a second audit found that roughly half the articles still failed the passage test, because each opened with a scene setting paragraph and the actual claims arrived several hundred words in.
The sequence that followed was therefore forced and correct: render first, restructure second, and only then think about which topics to add. Citations appeared within weeks of the render fix on the pages that already had usable passages, and the restructured half followed over the next two months.
Key takeaways
Three independent gates stand between your page and a citation: can a crawler fetch it, does the raw response contain your text, and does any passage make a complete claim alone. Test all three before writing anything new, because the cheapest wins are almost always at gate one or two. Check robots and bot protection explicitly per user agent, curl for a body sentence to catch client side rendering, and read 250 word windows in isolation to catch unusable passages. Then compare what could be cited against what actually is, because that gap is the work.
