If you run a membership site, a paid newsletter, or a course platform, AI crawlers create a problem other sites do not have: the content that makes you money is the content they want most, and it sits behind a login wall. The short answer is that authentication already protects you — Googlebot cannot log in, and neither can GPTBot or ClaudeBot. The real work is deciding what the public side of your site should look like to crawlers, so your free previews get found and cited while your paid content stays paid.
This guide covers the three layers that actually matter for paywalled sites: what login walls already do, how to use robots.txt without accidentally killing your search visibility, and how Google's paywalled-content structured data keeps your free previews indexed without looking like cloaking.
What a Login Wall Already Does
Start with the good news. A crawler that hits a login page gets a 401 or 403 response, or a redirect to a sign-in form. Google Search Central states plainly that if a server returns 401 or 403, Googlebot cannot access the URL and it will not be indexed. Googlebot cannot fill in forms or submit login credentials, and the same is true of AI crawlers. Your members-only lessons, your paid reports, your private community threads — none of that is reachable to a bot that cannot log in.
That means the common fear — "an AI crawler will scrape my whole course" — is mostly misplaced for properly gated content. The risk is not the gated content itself. The risk is everything around it: the free preview pages, the course landing pages, the public blog posts that summarise paid material, and the excerpts you publish to sell the membership. Those pages are public, crawlable, and exactly what a training bot will collect. Your rules should focus there.
There is one exception to watch for. If your site serves full content to any visitor whose user-agent looks like Googlebot — a practice sometimes used to get gated pages indexed — you have handed AI crawlers the same door, because many of them identify themselves honestly and some do not. Do not build a system that treats crawlers as logged-in users unless you understand exactly which crawlers you are letting in and what they do with the content. The safe default is: crawlers see what a logged-out visitor sees, nothing more.
The Robots.txt Rules for a Membership Site
Your robots.txt file has two jobs on a membership site. First, keep crawlers out of the low-value, logged-in areas so they do not waste requests on URLs that return a login page anyway. Second, block training crawlers from the public content you do want indexed. These are separate rules, and they sit in separate groups.
For the logged-in areas, block the paths that only exist behind a login. The exact paths depend on your platform, but the pattern is the same everywhere:
User-agent: *
Disallow: /account/
Disallow: /members/
Disallow: /dashboard/
Disallow: /checkout/
Disallow: /my-courses/
Disallow: /wp-admin/These rules are about crawl efficiency, not security. A robots.txt block does not hide a URL — the file is public, and anyone can read it — and it does not remove a page from search results. A blocked URL can still appear in Google as a bare link if other pages link to it. The login wall is what keeps the content private; robots.txt just tells crawlers not to bother knocking.
For the training crawlers, block them from the public side of the site while leaving search crawlers alone:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /That split keeps your public previews citable in ChatGPT and Claude answers — which is how many membership sites now get discovered — while keeping your free content out of training datasets. Build and check the file with the robots.txt generator and the robots.txt tester before it goes live.
The Mistake That Kills Your Search Traffic
The most damaging error on a membership site is also the most common: blocking Googlebot from the pages you actually want indexed. It usually happens in one of two ways.
The first is a blanket rule. Someone writes Disallow: / for all user-agents while testing a staging site, the file ships to production, and the whole site disappears from Google. The second is subtler: a rule that blocks a directory containing both private and public content. Blocking /courses/ hides the members-only lessons, which is fine, but it also hides the public course landing pages, the free sample lessons, and the curriculum pages that bring in new members. Googlebot cannot see those pages, so they cannot rank.
There is a related trap with noindex. A page that is blocked in robots.txt can never show Google its noindex tag, because Googlebot never fetches the page. If you want a page out of search results, keep it crawlable and put a noindex tag on it — do not block it in robots.txt. The rule of thumb: robots.txt controls crawling, noindex controls indexing, and authentication controls access. Use each tool for its own job.
Check your setup in Google Search Console after any change. The robots.txt report shows the file exactly as Google fetched it, and the coverage report will tell you if pages you expected to be indexed are suddenly excluded. Catching a bad rule in the first week saves months of lost traffic.
Paywalled Content and Google's Structured Data
Many membership sites want the best of both worlds: the full article indexed by Google, so it can rank and be found, while visitors who arrive from search see a preview and a paywall. Google supports this, but only if you declare it properly with structured data. Without the declaration, the gap between what Googlebot sees (the full article) and what a visitor sees (a paywall prompt) can look like cloaking, and cloaking can get a page demoted or removed from the index.
The mechanism is simple. On a paywalled page, you add structured data marking the page — or part of it — as not freely accessible, using the isAccessibleForFree property set to false, with a hasPart entry and a CSS selector pointing at the gated section. You serve Googlebot the full article, and the structured data tells Google explicitly that part of it is behind a paywall. That declaration is what separates a legitimate paywall from a cloaking attempt.
Google also provides two specialised crawlers for this: Googlebot Subscriber and Googlebot Registered User. If your paywall setup uses server-side checks to decide what a crawler sees, these are the user-agents to grant access to, so Google can index the full content while ordinary visitors still hit the paywall. The details depend on your platform, and the implementation belongs in your paywall plugin or server config, not in robots.txt.
One caution: the structured data must describe reality. If an entire article is free, do not label it as gated. If only the second half requires a subscription, mark only that section. Mislabelled paywall markup is worse than none, because it tells Google your pages are gated when they are not, and it trains the system to distrust your markup.
Platform Notes: WordPress, Ghost, and Course Platforms
On WordPress with a membership plugin, the gated content is usually protected by the plugin itself, and the plugin decides what a logged-out visitor sees. Your job is to make sure the plugin's preview — the free excerpt — is what crawlers see, and that the robots.txt file does not block the plugin's public pages. Most membership plugins have a setting for how much free content to show; set it deliberately rather than leaving the default.
On Ghost, membership and paid posts are a built-in feature. Public previews of paid posts are crawlable by default, and the full post is gated. The same rules apply: block trainers site-wide, keep search crawlers allowed, and do not block the preview URLs.
On course platforms such as Teachable, Kajabi, or Podia, you have less control. The platform decides what is crawlable and what is not, and you often cannot edit robots.txt at all. In that case, focus on what you can control: the public sales pages, the free lesson previews, and the blog. If the platform lets you add meta tags or custom code to public pages, use them to add noindex to pages you do not want in search, and use the platform's own settings to control what AI crawlers can see. Our platform-specific guides for Kajabi and Substack cover the details for two common cases.
What About AI Crawlers That Ignore Robots.txt?
Some crawlers ignore robots.txt, and some pretend to be browsers. No robots.txt file stops them. For a membership site, this is where the login wall earns its keep again: a scraper that ignores robots.txt still cannot log in, so the gated content stays gated. The exposure is limited to your public pages, which are public by definition.
For the public pages, your options are a web application firewall or a bot-management rule at the edge — for example, a Cloudflare rule that challenges or blocks requests matching known AI-crawler user-agents or IP ranges. That is a stronger layer than robots.txt, but it needs care: an over-broad rule can block legitimate visitors or the search crawlers you want to keep. Start with robots.txt, measure what actually arrives in your logs, and add edge blocking only for crawlers you can see ignoring your rules.
Watch your logs for a few weeks after publishing your rules. You should see GPTBot and ClaudeBot requests stop, while Googlebot and OAI-SearchBot continue. If a training crawler keeps arriving, note the user-agent and IP range, verify it with a reverse DNS check, and decide whether it deserves an edge rule. Our guide to verifying AI bots with reverse DNS explains how to tell a real crawler from an impersonator.
FAQ
Can AI crawlers access my members-only content?
No, if it is properly gated. Googlebot cannot log in or submit forms, and neither can AI crawlers. A login wall that returns 401 or 403 to unauthenticated visitors keeps the content out of every crawler's reach. The risk is your public previews and landing pages, which are crawlable by design.
Should I block Googlebot from my paywalled pages?
Usually no. Blocking Googlebot removes the page from search entirely, including the free preview that brings in new members. Use Google's paywalled-content structured data instead, so the full article is indexed while visitors see the paywall.
Does blocking a page in robots.txt remove it from Google?
Not reliably. A blocked page can still appear as a bare URL if other sites link to it. To remove a page from search results, keep it crawlable and add a noindex tag instead.
What is the difference between robots.txt and a login wall?
Robots.txt is a request that well-behaved crawlers honour; it is not security. A login wall is real access control: without credentials, nobody — human or bot — sees the content. Use robots.txt for crawl management and the login wall for privacy.
Can I let Google index my full article but show visitors a paywall?
Yes. Serve Googlebot the full article and mark the gated section with paywalled-content structured data, using isAccessibleForFree set to false. The declaration is what keeps this from looking like cloaking.
How do I stop AI crawlers that ignore robots.txt?
Use an edge layer such as a Cloudflare rule that blocks or challenges the crawler's user-agent or IP range. Check your logs first to confirm the crawler is actually ignoring your rules, and verify its identity with a reverse DNS lookup before blocking.
