How to Block AI Crawlers on Framer (2026): Robots.txt, Meta Tags & Cloudflare Workarounds

Yes, you can limit AI crawlers on a Framer site, but not the way you would on WordPress or a hand-coded site. Framer automatically generates and optimizes your robots.txt file, and it does not let site owners edit that file directly. So instead of pasting bot rules into robots.txt, you use two practical workarounds: per-page meta robots tags for pages you want kept out, and a Cloudflare or similar proxy layer to block named AI training bots before they reach your site. Keep Googlebot and Bingbot allowed, so normal search keeps working.

That distinction matters. This guide is not about hiding your whole site from search. It is about stopping training crawlers such as GPTBot and ClaudeBot from collecting your pages for model training, while leaving the search crawlers that bring you visitors alone.

Why Framer makes this trickier than other platforms

Framer is built for speed and simplicity. You design, you publish, and Framer handles much of the technical setup for you. That setup includes robots.txt. Framer creates the file, keeps it valid, and serves it at the usual address for your domain.

The trade-off is control. On many other platforms, you can open robots.txt and add a line for each bot you want to block. In Framer, there is no settings field for that file. You cannot upload a replacement, and you cannot add custom disallow rules to it through the normal editor or site settings.

That does not mean you are stuck. It means the control point moves. Instead of controlling bots at the robots.txt file, you control them at the page level with meta tags, or at the edge — the layer in front of your site — with a service like Cloudflare. Both routes work, and they solve slightly different problems. Understanding which one fits your goal will save you from blocking the wrong bots.

What Framer can't do — an honest list

Before you start, it helps to be clear about the limits, so you do not waste time looking for a setting that does not exist.

• Framer does not offer a direct robots.txt editor. You cannot add bot-specific disallow rules to the generated file.
• Framer does not offer a built-in per-bot toggle. There is no switch labeled for GPTBot, ClaudeBot, or any other AI crawler.
• Framer does not force a badly behaved bot to obey your wishes. Robots.txt itself is a request, not a lock, and the same is true of the workarounds in this guide. A crawler that ignores published rules can still try to fetch a public page.
• Framer cannot, on its own, tell the difference between a training bot and a search bot for you. You have to make that choice, bot by bot, and apply it through meta tags or an edge rule.

None of that makes Framer a bad choice for a content site. It just means your bot plan needs one extra layer. If you already use Cloudflare in front of your site, that layer may already be in place. If you do not, the meta-tag route is the fastest place to start.

First, know the bots: training crawlers vs. search and user bots

Not every bot with AI in its name does the same job. Some collect pages to train models. Others fetch a page so an AI tool can answer a user, or cite your page as a source. Blocking the first group is a common choice for site owners who do not want their work used for training. Blocking the second group can cost you visibility in AI answers and citations.

Here are the names to know, with the job each one is generally used for:

The key rule is simple: block training bots if that is your goal, but leave search bots alone unless you have a clear reason to hide from search as well. OAI-SearchBot and Claude-SearchBot power citations in AI answers. If you block them, your pages are less likely to be cited when someone asks a question your page answers well. That is a real cost, so make it a deliberate choice, not an accident from blocking every bot name you find in a list.

It is also worth repeating a point that trips up many site owners: robots.txt is a request, not a lock. Well-behaved crawlers from major providers generally honor it. A bot that ignores published rules will not be stopped by a robots.txt line alone. That is why the Cloudflare route matters — it can actually refuse a request, rather than just asking a bot not to make it.

Method 1: Use per-page meta robots tags in Framer

This is the route you can start today, inside Framer, with no extra service. It works best when you want to keep specific pages out, rather than blocking a single bot across your whole site.

A meta robots tag sits in the head of a page and tells crawlers how to treat that page. The classic controls are noindex, which asks search engines not to index the page, and nofollow, which asks them not to follow links from the page. Those controls apply broadly, not to one AI bot only, so use them with care.

That broad effect is both the strength and the limit of this method. If a page is a draft, a private client page, a duplicate test page, or content you simply do not want in any index, a meta robots tag is a clean, honest tool. If a page is a public article you want in Google Search but not in AI training data, a noindex tag is too blunt — it would also remove the page from Google Search. For that mixed goal, the Cloudflare route in Method 2 is the better fit, because it can block a named bot while leaving Googlebot alone.

Step-by-step: add a meta robots tag in Framer

1. List the pages first. Write down the exact pages you want to keep out, and why. Do not start with your whole site. A short, deliberate list is easier to test and easier to reverse.
2. Open the page settings in Framer. Go to the page you want to change, open its settings, and find the custom code area for that page. Framer lets you add custom code at the page level, which is where a head tag for that page belongs.
3. Add the meta tag for that page only. For a page you do not want indexed, add a meta robots tag with a noindex instruction. Keep the tag on that page only, not in a site-wide code block, unless you truly mean to apply it everywhere.
4. Publish the page. A saved but unpublished change does nothing for visitors or crawlers. Publish, then check the live page.
5. Check the live page source. Open the live page in a browser, view the page source, and confirm the meta tag is present in the head. If it is not there, it is not in the right code field, or the page was not published after the change.
6. Repeat for each page on your list. Do not copy the tag into a global header for convenience. Page-by-page control is the whole point of this method.
7. Keep a simple record. Note the page, the date you added the tag, and the reason. When a page later becomes public, or when you change your mind, that note tells you exactly what to remove.

After you add a noindex tag, do not expect the page to vanish from search results the same day. Search engines need to recrawl the page and process the new instruction. The tag is a request to search engines, and like robots.txt, it depends on the crawler honoring it. It is still the standard, documented way to keep a page out of an index, and it is the tool Framer gives you for page-level control.

One more caution: do not use nofollow as a substitute for noindex. They do different jobs. Nofollow is about links on the page. Noindex is about the page itself. If your goal is to keep the page out, noindex is the instruction that matters.

Method 2: Put the site behind Cloudflare and block AI bot user-agents

If your goal is to block named AI training bots across your whole site while keeping Google Search working, this is the stronger route. Cloudflare sits in front of your site and can inspect incoming requests, including the user-agent string that identifies the bot. You can create a rule that blocks requests whose user-agent contains a training bot name, and leaves everything else alone.

This is also the route that gets closest to actual enforcement. A blocked request at the edge does not reach your Framer site in the normal way. That is different from a robots.txt request, which a bot can ignore. It is not a perfect shield — a bot can change its user-agent, and no edge rule can stop every determined scraper — but for the named, well-known training bots, user-agent blocking is a practical and widely used control.

Step-by-step: set up Cloudflare bot blocking for a Framer site

1. Add your site to Cloudflare. Create a Cloudflare account if you do not have one, add your domain, and follow the setup steps to point your domain's name servers to Cloudflare. Your Framer site stays where it is; Cloudflare becomes the layer in front of it.
2. Confirm the site still loads. Before you add any blocking rule, visit your live site, click through a few pages, and confirm images and forms still work. Only add blocking rules after the basic setup is stable.
3. Create a firewall rule for training bots. In the Cloudflare dashboard, create a custom firewall or WAF rule. Set the condition to match the user-agent field, and match it against the bot names you chose to block — for example GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Applebot-Extended, and Bytespider. Use a contains-style match for each name, combined so that any one match triggers the rule.
4. Set the action to Block. For the training bot rule, choose Block, not a challenge. A challenge page can still be fetched by some automated clients, and it adds noise to your logs. A block is clearer and easier to test.
5. Do not add search bots to the block list. Leave Googlebot, Bingbot, OAI-SearchBot, and Claude-SearchBot out of this rule unless you have decided, on purpose, to give up that visibility. Double-check the rule before you save it. One wrong name here can hide you from search.
6. Save the rule and test it carefully. Use Cloudflare's own testing tools, or a simple request tool that lets you set a custom user-agent, to send a request with a blocked bot name and confirm it is blocked. Then send a normal browser request and confirm it loads. Then confirm that a Googlebot-style request is not blocked by your rule.
7. Watch the firewall events. For the first few days, check Cloudflare's firewall events for your new rule. You should see blocked requests from the bot names you listed, if those bots were visiting your site. If you see a legitimate visitor blocked, or a search crawler blocked, fix the rule right away.
8. Review the rule monthly. Bot names change, new training bots appear, and your own goals may change. A monthly check — five minutes, once a month — keeps the rule current without turning it into a daily chore.

Describe your rule in plain language in your own notes, for example: block any request whose user-agent contains GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Applebot-Extended, or Bytespider, and allow all other requests. That plain-language note is useful later, when you or a teammate need to change the rule and cannot remember why a name was added.

If you already use another WAF or proxy in front of Framer, the same idea applies there: match the user-agent, block the named training bots, allow search bots, test, and monitor. The tool name matters less than the logic of the rule.

Method 3: Plan your robots.txt, even though Framer writes it

You cannot paste rules into Framer's robots.txt, but planning the file is still useful. A planned robots.txt tells you exactly which bots you would block if you moved to a platform that allows editing, or if you add a proxy layer that serves its own robots.txt. It also stops you from making rushed, inconsistent choices in Cloudflare.

Use a free planning tool to draft the file you want. The robots.txt generator helps you build a clean file with the bots and rules you choose, and the robots.txt tester helps you check whether a given bot would be allowed or blocked by a file. Draft the file, test it against GPTBot, ClaudeBot, Googlebot, and Bingbot, and keep the result with your notes. When you build your Cloudflare rule, use the same bot list, so the two controls agree.

This step also helps you compare Framer with other builders. If you are weighing a move, or helping a client choose a platform, a side-by-side look at a similar guide — for example, how to block AI crawlers on Webflow — shows how much control each platform gives you over robots.txt and where the workarounds differ.

Common mistakes to avoid

• Blocking everything at once. A rule that blocks every bot name you can find will also block search and citation bots you probably want. Start with training bots only.
• Using noindex on public articles to stop training. Noindex also removes the page from search. Use it for pages you truly want out of every index, and use edge blocking for the training-only case.
• Adding the meta tag site-wide by accident. A global noindex tag hides your whole site from search. Always check whether a code field applies to one page or to the whole site before you save.
• Forgetting to publish. In Framer, an unpublished change is not live. After every meta-tag change, publish and verify on the live page.
• Assuming robots.txt is a lock. It is a request. For named bots that honor it, it works well. For bots that ignore it, you need edge blocking, and even then, determined scraping needs a broader defense.
• Never checking the rule again. Bot names and your content goals change. A rule you set in January and never review can quietly block something useful, or miss a new bot entirely.

Testing checklist after you make a change

1. Confirm the live page source shows your meta tag, if you used Method 1.
2. Confirm your Cloudflare rule blocks a test request with a training bot user-agent, if you used Method 2.
3. Confirm a normal browser request still loads the site.
4. Confirm a Googlebot-style request is not blocked by your rule.
5. Check Cloudflare firewall events over the next few days for blocked bot traffic and for any false blocks.
6. Recheck your key pages in search after a few weeks, to confirm you did not accidentally hide pages you want indexed.

If any check fails, undo the last change first, then work out why it failed. A small, reversible change is always better than a large rule you cannot explain.

What to do if a bot ignores your rules

First, confirm the facts. Check your logs to see which user-agent actually made the request, and whether your rule matched it. Many reports of ignored rules turn out to be a different bot name, a changed user-agent, or a rule that was saved but never turned on.

Second, check the official documentation for that bot. OpenAI publishes documentation for GPTBot, and Anthropic publishes documentation for ClaudeBot, including how each provider says its crawler can be blocked. If a provider's own documentation describes a control you have not used, add that control to your plan. For background, see OpenAI's bot documentation and Anthropic's guide to blocking ClaudeBot.

Third, tighten the edge rule rather than adding more robots.txt lines you cannot control in Framer. Edge blocking, rate limiting, and managed bot rules in Cloudflare are the levers that can actually refuse traffic. If abuse continues after those steps, it becomes a question for your hosting and security setup, not for a single page tag.

FAQ

Can I edit robots.txt directly in Framer?

No. Framer automatically generates and optimizes your robots.txt file, and it does not give site owners a direct editor for that file. Use per-page meta robots tags for pages you want kept out, and use Cloudflare or another proxy layer to block named AI training bots by user-agent.

Will blocking GPTBot or ClaudeBot hurt my Google ranking?

Blocking training bots alone should not hurt your Google ranking, because Googlebot is a different crawler. Problems start when a rule accidentally blocks Googlebot or Bingbot as well. Keep search bots out of your block list, test the rule after you save it, and check your search visibility after a few weeks.

Does a noindex meta tag stop AI training?

A noindex tag asks search engines not to index a page, and well-behaved crawlers generally honor it, but it is a blunt tool. It removes the page from search too. If you want a public article to stay in Google Search but not be used for training, edge blocking by bot name is a better fit than noindex.

Do I need a paid Cloudflare plan to block AI bots?

Basic custom firewall rules that match a user-agent are available on entry-level Cloudflare setups, and many site owners start there. Paid plans add managed bot rules and deeper controls, which can help on larger or heavily scraped sites. Start with a simple, tested custom rule, and only upgrade if you can point to a problem the free controls do not solve.

What if a bot ignores robots.txt and my Cloudflare rule?

First, verify the user-agent in your logs and confirm your rule actually matches it. Then check the provider's official crawler documentation for any additional control it offers. If the traffic continues, tighten your edge controls — blocking, rate limiting, and managed bot rules — and treat it as a security issue, not a robots.txt issue. No single tag can force a badly behaved bot to stop.

The bottom line for Framer owners

Framer trades direct robots.txt control for simplicity, and that trade shapes your bot plan. You cannot paste a block list into robots.txt, so do not spend time hunting for that setting. Instead, use meta robots tags for the specific pages you want out of every index, and use Cloudflare to block named training bots — GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Applebot-Extended, and Bytespider — while leaving Googlebot, Bingbot, and the search bots that cite your pages alone.

Write down your bot list, test each change on the live site, and review the rule once a month. That simple routine gives you real control over AI training crawlers on Framer, without hiding your best public pages from the search engines and AI answers that send you readers.