To block AI crawlers on Ghost, put a custom robots.txt file in your theme's root folder, then re-upload and activate the theme. Ghost serves that file in place of its default robots.txt, so any AI bot you disallow there will be told to stay out. If you cannot or do not want to edit your theme, use Cloudflare's bot controls in front of your site instead. For individual pages you want out of search results, add a noindex meta tag through Ghost's Code Injection. This guide covers all three methods, when to use each, and the mistakes that leave your content exposed.
Ghost is a clean, fast publishing platform, and its SEO defaults are good. But it has no plugin ecosystem like WordPress, so there is no "install a plugin and block GPTBot" button. Blocking AI crawlers on Ghost means working with the tools Ghost actually gives you: themes, Code Injection, redirects, and whatever sits in front of your site, such as Cloudflare. Once you understand those four levers, the job is straightforward.
How Ghost handles robots.txt by default
Every Ghost site serves a robots.txt file automatically at yoursite.com/robots.txt. If you have never customized it, that file is permissive: it allows all crawlers to visit all public pages. That is the right default for a new site that wants to be found. It becomes a problem only when you decide that some AI crawlers should not have free access to your work.
Ghost does not offer a robots.txt editor in the admin panel. There is no settings page where you can type rules and save them. Instead, Ghost lets your theme override the default file. If your active theme contains a file named robots.txt in its root folder, Ghost serves that file instead of its own default. That single mechanism is the foundation of AI-crawler blocking on Ghost, and it is officially documented by Ghost.
One important consequence: your custom robots.txt lives inside your theme. If you switch themes, the new theme's robots.txt — or Ghost's default, if the new theme has none — takes over. Your blocking rules do not travel with your site; they travel with your theme. Keep a copy of your robots.txt file outside the theme so you can restore it after a theme change, and re-check /robots.txt every time you switch themes.
Method 1: Custom robots.txt in your theme (the main method)
This is the method Ghost documents, and it works on both self-hosted Ghost and Ghost(Pro), because both let you upload themes. The steps are the same either way.
Step 1: Download your current theme. In Ghost Admin, go to Settings, then Design, then click "Change theme." Find your active theme and download it. You now have a zip file containing your theme. Keep an untouched copy of this zip as a backup before you change anything.
Step 2: Add your robots.txt file. Unzip the theme on your computer. In the theme's root folder — the same folder that contains package.json — create a plain-text file named exactly robots.txt. Write your rules in it. A sensible starting point that blocks AI training crawlers while leaving search engines and answer engines alone looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: *
Allow: /Each User-agent block names one crawler, and Disallow: / tells it to stay out of the whole site. The final block, User-agent: * with Allow: /, leaves everyone else alone. Adjust the list to your policy. If you want PerplexityBot to keep visiting because it sends you referral traffic, leave it out. If you want to block all AI crawlers, add the rest: OAI-SearchBot, Claude-Web, Google-Extended, Meta-ExternalAgent, Applebot-Extended, and the rest of the bots you have identified in your logs.
Step 3: Re-zip and upload the theme. Zip the theme folder back up, making sure the zip contains the theme files at its top level, not nested inside an extra folder. In Ghost Admin, go back to Settings, then Design, then "Change theme," and upload the new zip. Ghost will validate the theme; if it reports errors, fix them and re-upload. Activate the theme once it uploads cleanly.
Step 4: Verify the live file. Visit yoursite.com/robots.txt in a private browser window. You should see your custom rules, not Ghost's default. If you still see the default, your theme upload may not have activated, or a caching layer may be serving the old file — clear your cache and check again.
Step 5: Test the rules. Paste your robots.txt into a free robots.txt tester and check a few user-agents against a few URLs. Confirm that GPTBot is blocked from a sample post, and that Googlebot is still allowed. A typo in a user-agent name silently fails open: the bot you meant to block will not match the misspelled name, and it will crawl freely. Testing catches that before it costs you months of exposure.
If you are starting from scratch rather than editing an existing file, build the rules with our free robots.txt generator, which includes AI-crawler presets, then drop the result into your theme.
Method 2: Cloudflare rules (when theme edits are not practical)
Theme edits are the clean way to do this, but they are not always convenient. You may be on a managed Ghost setup without theme access, you may not want to re-upload a theme for a one-line change, or you may want enforcement that works even against bots that ignore robots.txt. That is where Cloudflare comes in.
Cloudflare sits in front of your site and can stop requests before they reach Ghost at all. That makes it stronger than robots.txt in one important way: robots.txt is a request that polite bots honor voluntarily, while a Cloudflare rule is enforced at the edge. A bot that ignores robots.txt still gets stopped by a firewall rule. The trade-off is that Cloudflare controls access for everyone, so a misconfigured rule can block real visitors or search engines. Test carefully.
Option A: Cloudflare's AI bot control. Cloudflare offers a one-click control for AI bots in its dashboard, under Security and then Bots. Turning it on blocks AI crawlers that Cloudflare has identified, without you having to list user-agents by hand. This is the simplest option if your site already runs behind Cloudflare, and it is a good first step while you set up the theme-based robots.txt file.
Option B: Custom WAF rules. For finer control, create a custom firewall rule that matches the user-agents you want to block. A rule that triggers on requests whose user-agent contains "GPTBot" or "ClaudeBot" and returns a block response will stop those crawlers regardless of what your robots.txt says. Use this for specific bots you have confirmed are ignoring your robots.txt file — check your server logs or Cloudflare analytics for their user-agents first, so you are blocking the exact tokens they actually send.
Option C: Rate limiting. Some crawlers are not worth blocking outright but should not be allowed to hammer your site. A rate-limiting rule that slows down aggressive crawlers protects your server without cutting off a bot that might send you traffic. This is especially useful for scrapers that rotate user-agents, since a rule based on request rate does not depend on the user-agent string being honest.
Cloudflare and robots.txt work best together. Robots.txt tells polite bots your policy in the standard way, and Cloudflare enforces it for the rest. Do not rely on Cloudflare alone: if you ever move off Cloudflare, your protection disappears with it, while a robots.txt file in your theme keeps working wherever the site is hosted.
Method 3: Code Injection for page-level noindex (a different job)
Ghost's Code Injection feature, under Settings and then Code Injection, lets you add code to your site's header or footer without editing theme files. You can use it to add a noindex meta tag to specific pages, which tells search engines not to include those pages in their indexes.
Here is the part most guides get wrong: noindex is not a crawler block. A page with noindex can still be crawled — the crawler fetches it, reads the tag, and then leaves it out of the index. If your goal is to keep AI crawlers from reading a page at all, noindex does not do that. If your goal is to keep a page out of search results while still letting crawlers read it, noindex is exactly right.
So when is Code Injection useful for AI-crawler control? Use it for pages that should not appear in search or AI answers but do not need to be hidden from crawlers: thin tag pages, author archives, internal landing pages, or pages you keep for navigation rather than for ranking. Add the tag to the Site Header injection field, ideally with a condition that limits it to the pages you mean, or add it to a specific page's code injection field in the post settings. Ghost also lets you set meta data per post in the post settings panel, which is the cleaner route for one-off pages.
Do not use noindex as a substitute for robots.txt blocking. And never combine the two on the same URL — a page blocked in robots.txt cannot be crawled, so the crawler never sees the noindex tag, and the page can linger in the index indefinitely. Pick one mechanism per URL: robots.txt to stop crawling, noindex to stop indexing.
Decide what to block before you block it
The biggest mistake Ghost site owners make is blocking every AI bot in one angry afternoon. AI crawlers are not all the same, and some of them send you traffic.
- Training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider) collect content to train models. They send no traffic. Blocking them is the lowest-risk decision most sites make.
- Search and answer crawlers (OAI-SearchBot, Claude-Web, PerplexityBot) fetch pages to answer user questions, and they can send referral visitors. Blocking them may cost you visibility in ChatGPT, Claude, and Perplexity answers.
- Search engine crawlers (Googlebot, Bingbot) power your traditional search rankings. Blocking them by accident — usually with a stray
Disallow: /underUser-agent: *— is the classic self-inflicted SEO disaster. Test before you publish. - Platform crawlers (Google-Extended, Applebot-Extended, Meta-ExternalAgent) are opt-out tokens for AI training by specific companies. Blocking them is a targeted training opt-out that leaves normal search crawling alone.
Write your policy down in one paragraph before you touch your theme: which crawlers you block, which you allow, and why. That paragraph becomes the comment block at the top of your robots.txt file and the basis for every future change. For a formal, machine-readable rights reservation that goes beyond robots.txt, see the W3C TDM Reservation Protocol. Without it, your rules accumulate into a contradictory mess that nobody on the team understands.
Verify that the block actually worked
Publishing a robots.txt file is not the same as blocking a bot. Verification has three parts. First, confirm the file is live and correct, as described in Method 1. Second, watch your logs. Ghost's own logs, your host's access logs, or Cloudflare analytics will show whether the bots you blocked have stopped requesting your pages. Give it a week — crawlers do not re-read robots.txt on every request, and some cache it for up to a day. Third, check for the bots that ignore robots.txt. If a blocked crawler keeps appearing in your logs at the same rate, it is ignoring your file, and that is when you escalate to a Cloudflare rule.
Also verify that you did not break anything. Check Google Search Console for a sudden drop in crawled pages, and confirm your most important posts are still indexed. A robots.txt change that blocks AI crawlers should not change your Google indexing at all. If it does, you blocked too much.
Mistakes Ghost owners make with AI crawlers
Editing the wrong file. Ghost has no admin robots.txt editor, so some owners edit a robots.txt file in a theme folder on their computer and forget to re-upload the theme. The live site keeps serving the old file. Always re-upload and activate.
Losing rules on a theme switch. Your robots.txt lives in your theme. A new theme means new robots.txt. Keep a standalone copy and re-apply it after every theme change.
Blocking everything with a wildcard. A Disallow: / under User-agent: * blocks Googlebot too. AI-crawler rules should name specific bots, and the wildcard block should stay permissive.
Confusing noindex with blocking. As covered above, noindex controls indexing, not crawling. Use the right tool for the job you actually want done.
Assuming robots.txt is enforced. Robots.txt is voluntary. Well-behaved bots follow it; scrapers and bad actors do not. Treat robots.txt as your published policy, and use Cloudflare or server rules for enforcement where it matters.
Forgetting llms.txt. Blocking is only half of AI-crawler control. The other half is guidance: an llms.txt file that points the bots you allow toward your best pages. If you block the trainers and guide the answer engines, you get the visibility without giving away the training data. Pair your robots.txt work with a hardened server configuration if you run Ghost behind Apache, and re-test after every theme update.
FAQ
Does Ghost have a robots.txt editor in the admin panel?
No. Ghost generates a default robots.txt automatically, and the way to customize it is to include a robots.txt file in your theme's root folder, then upload and activate that theme. Ghost then serves your file instead of its default.
Can I block AI crawlers on Ghost(Pro) without server access?
Yes. The theme method works on Ghost(Pro) because you can upload custom themes there. If you do not want to edit your theme, put Cloudflare in front of your site and use its AI bot controls or a custom firewall rule.
Will blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI's training crawler and has no role in Google Search. Googlebot is a separate crawler. As long as your robots.txt allows Googlebot, blocking GPTBot has no effect on your Google rankings.
Should I use noindex or robots.txt to keep a page away from AI crawlers?
Use robots.txt to stop crawlers from fetching a page. Use noindex to keep a page out of search indexes while still letting crawlers read it. Never put both on the same URL, because a robots.txt block prevents the crawler from ever seeing the noindex tag.
How long does it take for a robots.txt change to take effect on Ghost?
The file itself updates as soon as you activate the new theme. Crawlers may cache the old file for up to a day, and some bots only re-check it periodically, so allow a week before judging whether a blocked bot has actually stopped visiting. Check your logs to confirm.
Do I need to re-apply my robots.txt rules after changing themes?
Yes. Your custom robots.txt lives inside your theme, so switching themes replaces it. Keep a standalone copy of your robots.txt file and add it to every new theme before activating it.
