Cloudflare's Pay Per Crawl lets publishers charge AI crawlers a fee every time they fetch a page. When a crawler requests a priced page without offering payment, Cloudflare's edge answers with an HTTP 402 — the web's long-dormant "Payment Required" status — carrying your price. The AI company either pays the toll or walks away. It launched as a private beta in July 2025, and as of October 2026 it is still in private beta, with a sibling program called Pay Per Use now in testing. This guide explains exactly how it works, what it costs, what you could realistically earn, and the honest limits nobody in the launch coverage mentions.
How Pay Per Crawl Actually Works
The mechanism is deliberately simple. You set a price for your content — one default price for your domain, with the option to vary it by page. When an AI crawler requests a page, it can signal its willingness to pay in its request. If it does, the page is served normally with an HTTP 200. If it doesn't, Cloudflare returns HTTP 402 Payment Required along with your pricing information. Because Cloudflare sits between the crawler and your server, a crawler that refuses to pay is actually stopped — this is not a voluntary signal like robots.txt.
The business plumbing matters as much as the status code:
- Cloudflare acts as the merchant of record. It handles enrollment, usage records, billing, and payment — you don't negotiate with AI companies one by one.
- Minimum price is $0.001 per successful retrieval. Publishers choose their default price and can use dynamic pricing for different content. At the minimum, 1,000 paid retrievals generate $1 in gross charges, before any applicable deductions.
- Payouts run through Stripe Connect. Publishers create a dedicated Cloudflare Stripe Connect account, and payouts are monthly to publishers in good standing, subject to settlement periods and minimum thresholds.
- It builds on AI Crawl Control, Cloudflare's dashboard for monitoring AI traffic, which is available on all plans and lets you allow or block crawlers by category.
Where It Stands in October 2026
Two developments in the last two months changed the picture significantly.
September 15, 2026: the default flip. Cloudflare began default-blocking AI training and agent crawlers on ad-supported pages for new domains, new sites set up by existing customers, and existing free customers. This was not a payment product — it was a blocking default — but it established the principle that the infrastructure itself, not just individual site owners, gets a say. Cloudflare's system distinguishes between AI Search bots and AI training crawlers, which puts pressure on AI companies to split mixed-use bots into separate, honestly-declared identities.
September 30, 2026: Pay Per Use beta. Cloudflare launched a beta that the company describes as shaping Pay Per Crawl into Pay Per Use. The difference: Pay Per Crawl charges for access (the fetch), while Pay Per Use pays for what happens next — for example, when your content is surfaced inside an AI-generated answer. The roles are reversed from Pay Per Crawl: each AI company defines the use it will pay for and sets a price, publishers decide which offers to accept, and the AI company self-reports usage (timestamp, source URL, event ID). Cloudflare checks reported uses against enrolled publishers, then bills the buyer and pays publishers monthly. Early partners named in coverage include Ceramic.ai and You.com. No buyers were named at launch, and Cloudflare has been careful to call it an experiment.
One caution from careful coverage: don't collapse these into a tidy "pay per answer" story. Pay Per Crawl bills per request, not per answer. Pay Per Use is Cloudflare's phrase for a direction, and the concrete per-answer examples so far are partner arrangements, not a settled product. Nobody in this market has agreed on the billable unit yet.
The Money Math: What Could You Actually Earn?
Let's do the arithmetic the launch posts skip. At Cloudflare's $0.001 minimum, you need 1,000 paid crawler retrievals to gross $1. A mid-size blog getting 50,000 AI crawler hits a month — every one of them paid at minimum price — would gross $50 a month. Price your deep research at $0.05 a retrieval and 10,000 paid fetches gross $500. Those numbers are before deductions and thresholds, and they assume the crucial thing: that crawlers actually pay.
That assumption is the whole game. A toll only earns when a crawler wants your specific pages enough to pay, and that want tracks the same thing that earns you a citation: being a distinct, trusted source on a subject. A toll booth on a road nobody drives collects nothing. The publishers most likely to earn are the ones AI companies already crawl heavily — news, reference, and data sites — because their content is what the crawlers are already fetching for free.
Cloudflare's CEO has cited stark crawl-to-referral ratios to make the case for publisher harm: 18 pages crawled per referral for Google, 1,500 to 1 for OpenAI, and 60,000 to 1 for Anthropic. Those ratios describe the scale of uncompensated fetching, not a revenue forecast — but they explain why a per-crawl toll appeals to publishers who see bots consuming thousands of pages for every visitor sent back.
The Honest Limits
Pay Per Crawl is a real mechanism with real limits. Here they are, plainly:
- It is still a private beta. Cloudflare's own documentation describes it as a closed beta more than a year after launch. You apply for early access; you don't just switch it on.
- Demand is unproven at small scale. AI companies pay for content they need. A niche blog with thin crawler traffic has little leverage, whatever price it sets. As one publisher-economics analysis put it, the honest limit is demand.
- It will not replace ad revenue. Cloudflare's own partnerships executive said in an October 2026 interview that publishers should not expect AI payments to replace ad dollars one-for-one. Treat this as a new line item, not a rescue plan.
- Usage reporting relies on the buyer. In the Pay Per Use model, the AI company reports its own usage. The program rules require complete reporting and Cloudflare verifies matches, but self-reporting is inherently weaker than edge-measured billing.
- Charging can cost you citations. A crawler that declines your price stops fetching — which also means it stops citing you. Set the toll with your AI-visibility strategy in mind, not just your revenue hopes.
None of this makes Pay Per Crawl a bad idea. It makes it a pricing decision, like any other: what is access to your content worth, to whom, and what do you lose by charging?
Pay Per Crawl vs. the Alternatives
Cloudflare is not the only tollbooth being built. Knowing the alternatives helps you price and position correctly:
- robots.txt blocking (free). Costs nothing, stops polite crawlers, earns nothing. The right choice when you object to the use itself, not just the price. Our universal AI crawler template covers every major training bot, and our Cloudflare bot-blocking guide walks through the dashboard controls.
- Really Simple Licensing (RSL). Launched September 2025 and backed by publishers including Reddit, Yahoo, Medium, O'Reilly Media, Ziff Davis, and People Inc., RSL extends the robots.txt idea: instead of yes/no, sites publish licensing terms supporting free, attribution, subscription, pay-per-crawl, and pay-per-inference models. Its backers describe it as working alongside Cloudflare's enforcement rather than replacing it — and it still depends on AI companies cooperating.
- TollBit. A standalone service where publishers set rates and TollBit's proxy logs each paid request on a ledger both sides can see. Free for publishers; AI companies pay a fee.
- AWS WAF's x402 support. In June 2026, AWS added the same idea to its Web Application Firewall: AI crawler requests to protected content can return HTTP 402, with price and payment details riding along through the x402 protocol as a machine-readable manifest, settling in stablecoin. Two of the largest infrastructure providers now sell a tollbooth for machines.
- Google's contribution pilot. Google is running a small pilot paying publishers for content used in AI Overviews, AI Mode, and the Gemini app — reportedly around 100 sites, some paid under $1,000 over several months, with no published methodology. Invitation-only, not a product you can join.
- Direct licensing deals. The $250-million-scale agreements between AI labs and big publishers. Not available to the rest of us — which is exactly the gap Pay Per Crawl tries to fill.
How to Get Started Today
If the model fits your site, here is the practical path:
- Measure first. Before pricing anything, find out how much AI crawler traffic you actually get. Check your server logs or CDN analytics for user agents like GPTBot, ClaudeBot, and PerplexityBot. If the volume is trivial, a toll earns trivial money — block or allow instead, and move on. Our guide to AI bots and AdSense revenue covers the traffic-quality side of this decision.
- Apply for the Pay Per Crawl private beta through Cloudflare's product page. While you wait, enable AI Crawl Control on your zone and categorize your crawler traffic — block, allow, or (once enrolled) charge.
- Set tiered prices. Commodity pages at or near the $0.001 minimum; genuinely unique assets — original research, proprietary data, deep guides — priced higher. Dynamic pricing exists so you can do exactly this.
- Set up Stripe Connect payouts per Cloudflare's documentation, and note that the dashboard may not yet display accrued balances — publishers currently need to request balance updates.
- Watch what the toll does to your AI visibility. Track citations and referral traffic from AI answers alongside the revenue. If charging a bot costs you more in lost visibility than it earns, the price is wrong.
What Charging Means for Your AI Visibility
This is the strategic point the launch posts leave out: the moment you can set terms, the terms become a visibility tradeoff. A crawler that pays your toll keeps reading and keeps citing you. A crawler that declines stops — and your pages fade from that AI company's answers. Price training crawlers (which give you nothing back) aggressively, and think twice before tolling the answer-engine bots that send you cited traffic. The publishers who will do best here are the ones who treat Pay Per Crawl as part of their AI-search strategy, not as a separate revenue gimmick.
For the blocking side of that strategy, our free robots.txt generator builds precise per-bot rules, and the pros-and-cons guide to blocking AI crawlers helps you decide which bots belong on which side of the toll.
FAQ
Is Cloudflare Pay Per Crawl generally available?
No. It launched as a private beta in July 2025, and Cloudflare's documentation still describes it as a closed beta. Publishers apply for early access. The related Pay Per Use program entered beta on September 30, 2026, also with limited participation.
How much does it cost to use Pay Per Crawl?
Publishers set their own prices starting at $0.001 per successful retrieval, with dynamic pricing per page. Cloudflare acts as the merchant of record and handles billing; payouts go out monthly via Stripe Connect, subject to settlement periods and minimum thresholds. Treat published fee details as subject to change and check Cloudflare's current payout documentation before enrolling.
Will charging AI crawlers hurt my Google rankings?
No — Pay Per Crawl targets the AI crawlers you choose to price, not Googlebot. Your Google Search crawling and rankings are unaffected. The visibility question is about AI answers, not classic search: a crawler that declines your price stops fetching your pages, which can reduce your citations in that company's AI products.
What happens when a crawler refuses to pay?
It gets an HTTP 402 Payment Required response with your pricing, and it does not get the content. Unlike robots.txt, which relies on the crawler's goodwill, the toll is enforced at Cloudflare's edge — a non-paying crawler is actually stopped.
Does Pay Per Crawl replace ads.txt or AdSense?
No. It is a separate revenue stream for AI crawler access, not a substitute for display advertising. Cloudflare's own team cautions against expecting AI payments to replace ad revenue dollar for dollar. Keep your ads.txt and AdSense setup exactly as it is, and treat crawler tolls as additive.
Can I charge different prices for different pages?
Yes. You set one default price for your domain and can vary it by page with dynamic pricing. The sensible pattern is minimum pricing for commodity content and higher prices for original research, proprietary data, and other content AI companies can't easily get elsewhere.
Bottom line: Pay Per Crawl is a real, working tollbooth for AI crawlers — HTTP 402 at the edge, you set the price from $0.001 up, Cloudflare handles billing. It is still in private beta, it only earns when crawlers actually want your pages enough to pay, and it won't replace ad revenue. Use it as a pricing decision inside your AI-visibility strategy: toll the training crawlers freely, think hard before tolling the answer engines that cite you, and measure both the revenue and the citations.
