AI indexing is already being monetized directly, in early and fragmented forms, and the shape of the market is becoming clear. On the supply side, publishers can now charge AI crawlers per request through Cloudflare’s pay per crawl, declare machine-readable licence terms through the RSL standard, and share in subscription revenue such as Perplexity’s Comet Plus. On the demand side, Google sells ads inside AI Overviews and AI Mode. What does not exist, and is unlikely to arrive in a recognisable form, is a way to pay for the citation itself, because an engine that sold its answers would lose the trust that makes them worth reading. For most businesses that means the valuable position stays free to earn: being the source the answer cites. SQSEO tracks that position, the share of AI answers that name you, because it is still won with content rather than a budget.
Why the old bargain is breaking
For two decades the web ran on a simple exchange. Publishers let search engines crawl their pages for free, and search engines sent visitors back through links. Advertising on the publisher’s site and on the search results page paid for both sides.
AI answers strain that exchange. A system that reads pages and writes a complete answer may send far fewer visitors back, and the visitors it sends may arrive later in their journey. Publishers whose income depends on page views have noticed, and their response is visible in robots.txt files across the web.
The clearest measurement is the study Consent in Crisis, an audit of around 14,000 web domains by the Data Provenance Initiative. It found that in a single year, from 2023 to 2024, more than 5% of all tokens in the widely used C4 training corpus became fully restricted by robots.txt, and that 28% or more of the most actively maintained, critical sources in C4 were fully restricted. Terms of service restrictions covered an even larger share. Blocking is the first tool publishers reached for. Charging is the second.
What forms of monetization already exist?
| Mechanism | Who pays whom | How it works | Status |
|---|---|---|---|
| Cloudflare pay per crawl | AI crawler pays the site | Site sets a price per request; crawlers get HTTP 402 and can agree to pay | Private beta, announced July 2025 |
| RSL licensing standard | AI company pays the publisher or a collective | Machine-readable licence terms in robots.txt, headers, feeds; pay per crawl or per inference | RSL 1.0 specification published |
| Direct licensing deals | AI company pays a large publisher | Negotiated contracts for access to archives and fresh content | Many bilateral deals, terms mostly private |
| Revenue sharing subscriptions | Users pay the engine, engine shares with publishers | Perplexity Comet Plus passes subscription revenue to participating publishers | Launched August 2025 |
| Ads in AI answers | Advertisers pay the engine | Labelled ads shown in or below AI Overviews and AI Mode | Expanding since Google Marketing Live 2025 |
Pay per crawl
Cloudflare’s pay per crawl, announced on 1 July 2025, gives sites three options per AI crawler: allow, charge or block. A site sets one price per request for the whole domain. A crawler that requests a charged page receives an HTTP 402 Payment Required response with the price and can retry while agreeing to pay; Cloudflare handles billing and pays the publisher. The feature is in private beta. Its importance is less the money than the precedent: it revives a long-dormant status code to put a price on machine access.
Licensing standards
RSL, Really Simple Licensing, is an open standard that lets publishers declare licence terms for AI use in robots.txt, web pages, HTTP headers, media files and RSS feeds. Terms can require attribution, payment per crawl or payment per inference. A nonprofit RSL Collective can negotiate on behalf of many publishers, and supporters listed on the site include Cloudflare, Fastly, Akamai, Reddit, Yahoo and Ziff Davis. It turns the robots.txt file from a yes or no into a price list.
Revenue sharing
Perplexity’s Comet Plus, announced on 25 August 2025, is a $5 subscription, included in its Pro and Max plans, whose revenue Perplexity says it distributes to participating publishers minus a small portion for compute. It is a different model again: users pay, and the engine passes the money to sources.
Ads in AI answers
On the demand side, Google said at Google Marketing Live 2025 that it was expanding ads in AI Overviews to desktop and bringing ads to AI Mode. These are labelled ads placed in and below the AI response, bought through existing campaign types. They monetize the answer page, not the index, and they do not change which sources the answer cites.
Will brands be able to pay for citations?
This is the question most marketers actually mean, and the answer is probably not, at least not openly.
An AI answer is valuable because users believe it reflects the best available information. Research on generative search already shows how fragile that trust is: the verifiability study of generative search engines found that cited sources often did not fully support the statements attached to them. An engine that let advertisers buy citations would make that problem deliberate, and users would discount every answer.
What is more likely is the pattern search followed: clearly labelled sponsored placements beside or below the answer, with the organic answer kept separate. That leaves the organic citation as the position brands compete for with content, in the same way organic rankings stayed separate from search ads.
How should publishers decide whether to charge?
The decision depends on whether your content is your product or your marketing.
If the content is the product, news, research, reference, recipes, reviews, then AI systems using it without sending traffic is a direct threat to revenue. Charging, licensing or blocking training crawlers is a rational response, and the new tools give you finer control than an outright block.
If the content is marketing, product pages, guides, documentation, comparison pages, then AI answers are a distribution channel. Blocking or charging retrieval crawlers removes you from the answers buyers now read, and the competitor who stays open takes the mention. Our guide on whether robots.txt blocks prevent ChatGPT scraping explains what blocking does and does not achieve.
Most businesses sit in the second group, even if they publish a lot of content. Their articles exist to be found and acted on.
Training crawlers versus retrieval crawlers
The single most useful distinction in this debate is between crawlers that collect data to train models and crawlers that fetch pages to answer a live question. Several AI companies run them separately, with different user agents, precisely so sites can treat them differently.
Blocking a training crawler keeps your content out of future model training. Blocking a retrieval or search crawler keeps your pages out of live answers, which is usually where the business value is. Many sites that intended the first have accidentally done the second. Our comparison of blocking ClaudeBot versus appearing in Claude’s recommendations walks through the difference for one provider, and the same logic applies to the others.
A sensible default for a marketing site is to allow retrieval crawlers, decide deliberately about training crawlers, and review the list every few months, because new user agents appear regularly.
How do you see which AI crawlers already visit your site?
Before deciding to charge, block or welcome anyone, find out who actually comes. Most sites have never looked, and the answer is often surprising.
Start with your server or CDN logs. Filter requests by user agent and look for the AI crawlers by name; each major provider documents its user agents. Count requests per crawler per week and note which sections of the site they read. A retrieval crawler fetching your pricing and comparison pages is a sign that your content is being used to answer buying questions. A training crawler working through your entire archive is a different kind of visitor.
Then check what your robots.txt actually says. Many sites inherited rules from a plugin, a template or a hurried decision during the first wave of AI news, and block crawlers they never meant to block. Compare the rules against the crawlers you found in the logs.
Finally, look for the outcome. Referral traffic from AI assistants shows up in analytics, usually as referrals from the assistant’s domain, and sometimes as direct traffic. If a crawler visits often and its assistant sends visitors who convert, it is a distribution partner, whatever the debate about payment says.
This audit takes an afternoon and turns an abstract policy question into a concrete one: which of these specific visitors is worth more to us open than closed?
What does this mean for small publishers?
Small publishers face the same economics as large ones with far less leverage. A national newspaper can negotiate a licensing deal; a niche blog with a few thousand loyal readers cannot. That is why the collective approaches matter: pay per crawl at the network level and collective licensing through standards such as RSL are the first mechanisms that let a small site set a price without a negotiating team.
For a small publisher whose income comes from advertising or affiliate links on its own pages, the honest calculation compares two numbers: the referral traffic AI assistants send today, and the plausible income from charging. For most small sites in 2026 both are small, and the referral traffic is easier to grow. Blocking everything is rarely the answer; deciding crawler by crawler, with the logs open, usually is.
What this means for SEO and GEO work
If crawling becomes partly paid, the open web available to AI systems shrinks, and the pages that remain freely readable become a larger share of what engines can cite. For businesses that stay open, that is an opportunity: less competition in the citation pool for commercial questions.
It also raises the value of being cited relative to being clicked. The GEO study found that adding statistics, quotations and citations to content improved visibility in generative answers by up to 40% in its experiments, which shows that how a page is written measurably changes whether it is used. That work does not get more expensive when crawling is monetized. If anything it gets more valuable, because fewer competitors will be readable.
Measuring that position is the job SQSEO does: which of your target questions you appear in, how often across repeated runs, and which competitors are named instead. As monetization spreads, that free organic share becomes a clearer asset to track, and since prompt volumes are estimates, share of answers is the more honest number to report.
What happens next?
A few developments look likely over the next year or two, based on where the mechanisms already point.
More crawler identification. Charging only works if crawlers identify themselves reliably, so expect stronger verification of AI user agents and penalties for crawlers that disguise themselves.
Collective bargaining. Small publishers have little leverage alone. Collectives such as the one RSL proposes let thousands of sites negotiate together, much as music rights organisations do.
Separate prices for training and retrieval. Retrieval sends at least some attribution and traffic; training does not. Pricing will likely reflect that.
More ad inventory around answers. As usage shifts toward AI answers, advertising follows, in clearly labelled formats.
None of these changes the basic strategy for a business site: be readable by the systems your buyers use, answer their questions better than anyone else, and measure whether you are named.
Key takeaways
AI indexing is already monetized in early forms: Cloudflare pay per crawl and the RSL standard let publishers price machine access, Perplexity’s Comet Plus shares subscription revenue with publishers, and Google sells labelled ads in AI Overviews and AI Mode. Publishers restricted access fast, with more than 5% of C4 tokens fully restricted within a year. Paid citations are unlikely, because selling answers would destroy trust in them. Charge or block if your content is your product; stay open to retrieval crawlers if your content is your marketing, and distinguish training crawlers from retrieval crawlers before changing anything.
