To block AI crawlers on Webflow, go to Project Settings, open the SEO tab, and add your rules in the robots.txt field. Then save, publish your site, and check that the live file at yourdomain.com/robots.txt shows your new rules. Webflow has no plugin system and no server access, so robots.txt is your main blocking tool. For individual pages, use the per-page setting to exclude them from search engine indexing. This guide walks through all three methods, what to block, and how to verify it worked.
Webflow sits between two worlds. It is a visual builder, so you do not touch a server. But unlike simpler builders, it gives you real SEO controls: a custom robots.txt field, per-page indexing settings, redirects, and password protection. That is enough to run a solid AI-crawler policy. What you cannot do is enforce rules at the server level. There is no .htaccess file to edit and no firewall panel inside Webflow. So your strategy has two layers: publish a clear robots.txt policy that polite bots follow, and use Webflow's page-level controls for the pages that need stronger handling.
Before you block anything: decide your policy
The most common mistake is blocking every AI bot in one go and then wondering why referral traffic dropped. AI crawlers are not one thing. They fall into groups, and each group deserves its own decision.
- Training crawlers collect content to train models. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and Bytespider (ByteDance) are the best known. They do not send you traffic. Blocking them is the lowest-risk decision most sites make.
- Answer-engine crawlers fetch pages to answer user questions. OAI-SearchBot, Claude-Web, and PerplexityBot fall in this group. They can send referral visitors when your pages get cited. Blocking them may cost you visibility inside ChatGPT, Claude, and Perplexity answers.
- Search engine crawlers power traditional search. Googlebot and Bingbot belong here. Never block them by accident. A stray
Disallow: /underUser-agent: *is the classic self-inflicted SEO disaster. - Opt-out tokens are special user-agents that companies offer for training opt-outs. Google-Extended (Google), Applebot-Extended (Apple), and Meta-ExternalAgent (Meta) let you opt out of AI training while leaving normal search crawling alone.
Write your policy down in one paragraph before you touch any setting: which crawlers you block, which you allow, and why. That paragraph becomes the comment at the top of your robots.txt file and the reference for every future change. Without it, rules pile up into a contradictory mess nobody on the team understands.
Method 1: Custom robots.txt in Project Settings
This is the main method on Webflow, and it is officially supported. Webflow serves a default robots.txt file for every site. When you add your own rules in Project Settings, Webflow serves your version instead.
Step 1: Open Project Settings. From your Webflow dashboard, open the project you want to change and go to Project Settings. Click the SEO tab, then scroll to the Indexing section. You will see the robots.txt field there.
Step 2: Write your rules. A sensible starting point blocks AI training crawlers while leaving search engines and answer engines alone:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /Each User-agent block names one crawler, and Disallow: / tells it to stay out of the whole site. The final block, User-agent: * with Allow: /, leaves everyone else alone. If you want PerplexityBot to keep visiting because it sends you referral traffic, leave it out. If you want to block answer-engine crawlers too, add OAI-SearchBot, Claude-Web, and PerplexityBot to the list. If you are starting from scratch, build the rules with a free robots.txt generator that includes AI-crawler presets, then paste the result into the field.
Step 3: Save and publish. Click Save Changes, then publish your site. Editing the field alone does not change the live file. Your site must be published for the new robots.txt to go live. This is the step people forget most often.
Step 4: Verify the live file. Open yourdomain.com/robots.txt in a private browser window. You should see your custom rules. If you still see the default file, the publish may not have completed, or a caching layer may be serving the old version. Republish and check again.
Step 5: Test the rules. Paste your robots.txt into a free robots.txt tester and check a few user-agents against a few URLs. Confirm that GPTBot is blocked from a sample page and that Googlebot is still allowed. A typo in a user-agent name fails silently: the bot you meant to block will not match the misspelled name, and it will crawl freely. Testing catches that before it costs you months of exposure.
Two plan notes. First, the custom robots.txt field is available on paid site plans with a connected custom domain. If you are on a free plan or a webflow.io staging subdomain, you will not see the field. In that case, use the global toggle to disable search engine indexing on the staging site instead, and set up your real robots.txt when you move to a paid plan. Second, Webflow's own sitemap is generated automatically. Keep the sitemap line in your robots.txt pointing at your live domain so crawlers that read robots.txt find it right away.
Method 2: Page-level noindex for pages you want out of search results
Robots.txt controls crawling for the whole site. Sometimes you need finer control over one page. Webflow gives you a per-page setting for that. Open the page in the Designer, open its Page Settings (the gear icon), and look under SEO settings for the option to exclude the page from search engine indexing. Checking it adds a noindex directive to that page.
Here is the part most guides get wrong: noindex is not a crawler block. A page with noindex can still be crawled. The crawler fetches it, reads the tag, and then leaves it out of the index. If your goal is to keep AI crawlers from reading a page at all, noindex does not do that. If your goal is to keep a page out of search results while still letting crawlers read it, noindex is exactly right.
Use page-level noindex for pages that should not appear in search or AI answers but do not need to be hidden from crawlers: thin utility pages, thank-you pages, internal landing pages, duplicate or near-duplicate pages, and pages you keep for navigation rather than ranking. Do not use noindex as a substitute for robots.txt blocking. And never combine the two on the same URL. A page blocked in robots.txt cannot be crawled, so the crawler never sees the noindex tag, and the page can linger in the index indefinitely. Pick one mechanism per URL: robots.txt to stop crawling, noindex to stop indexing.
Webflow also has a site-wide toggle to disable search engine indexing entirely. That is the right tool for a staging site or a site under construction. It is the wrong tool for a live site, because it hides you from Google and Bing too. Never leave it on by accident after launch. If your pages suddenly stop appearing in search results, this toggle is one of the first things to check.
Method 3: Protect staging sites and private pages
Most Webflow sites have a staging version on a webflow.io subdomain. Staging sites are a common leak: AI crawlers find them, index draft content, and sometimes rank the staging copy above the real site. Protect staging before it becomes a problem.
For the staging subdomain, turn on the setting to disable search engine indexing. That keeps search engines and well-behaved AI crawlers away from your drafts. If the staging site contains anything sensitive, go further and password-protect it. Webflow supports password protection at the site level and for individual pages. A password prompt stops every crawler, polite or not, because a bot cannot log in. Use password protection for client previews, pre-launch content, and any page that should never be public.
The same logic applies to private pages on your live site: membership content, client-only pages, and internal tools. A robots.txt rule asks bots to stay out. A password actually keeps them out. When the content matters, prefer the stronger control.
What Webflow cannot do (and how to work around it)
Honest limits matter. Webflow gives you no server access, so there are three things you cannot do inside Webflow itself.
You cannot edit .htaccess or server config files. On an Apache server, you could block bots at the server level with .htaccess rules. Webflow does not expose that layer. Your robots.txt file is a published policy, not an enforced barrier. Well-behaved bots follow it. Scrapers and bad actors do not.
You cannot run a firewall or bot-management rule inside Webflow. There is no Webflow-native panel for blocking a user-agent at the edge. If you need enforcement against a bot that ignores your robots.txt file, you need something in front of Webflow, such as a CDN or edge layer you control. That is an advanced setup: proxying in front of Webflow can interfere with SSL and publishing if it is misconfigured, so only take this route if you already run that infrastructure and can test it carefully.
You cannot see full server logs. Webflow does not hand you raw access logs, so you cannot grep for a bot the way you would on a self-hosted site. Instead, watch Webflow's site analytics for traffic patterns, and use Google Search Console to confirm your pages are being crawled and indexed normally. If you run an edge layer in front of Webflow, its logs become your best view of bot traffic.
None of these limits make Webflow a bad choice for AI-crawler control. They just mean your plan should lean on the tools Webflow actually gives you: a clean robots.txt policy, page-level indexing settings, password protection, and a habit of verifying after every change.
Do not forget the other half: guide the bots you allow
Blocking is only half of AI-crawler control. The other half is guidance. The bots you allow still need to find your best pages, and an llms.txt file does exactly that: it points AI systems toward your most important pages in a format they can read quickly. If you block the trainers and guide the answer engines, you get the visibility without giving away the training data.
For a Webflow site, you can add an llms.txt file as a static asset or a plain page at the site root, then link to it from your key pages. Keep it short and honest: your main pages, your best guides, and your contact and about pages. Do not stuff it with every URL on the site. A short, accurate llms.txt file beats a long, stale one.
Verify that the block actually worked
Publishing a robots.txt file is not the same as blocking a bot. Verification has three parts. First, confirm the file is live and correct, as described in Method 1. Second, confirm you did not break anything: check Google Search Console for a sudden drop in crawled pages, and confirm your most important posts are still indexed. A robots.txt change that blocks AI crawlers should not change your Google indexing at all. If it does, you blocked too much.
Third, watch for the bots that ignore robots.txt. Give it a week. Crawlers do not re-read robots.txt on every request, and some cache it for up to a day. If a blocked crawler keeps showing up in your analytics or edge logs at the same rate, it is ignoring your file. That is when you escalate to password protection for the affected pages or an edge-layer block, instead of adding more robots.txt lines that the bot will also ignore.
Mistakes Webflow owners make with AI crawlers
Forgetting to publish after editing robots.txt. The field saves, but the live file only changes when you publish. Always publish, then check the live file in a private window.
Blocking everything with a wildcard. A Disallow: / under User-agent: * blocks Googlebot too. AI-crawler rules should name specific bots, and the wildcard block should stay permissive.
Combining robots.txt blocks with noindex on the same URL. As covered above, a robots.txt block prevents the crawler from ever seeing the noindex tag. Pick one mechanism per URL.
Leaving staging sites open. A webflow.io subdomain with indexing enabled is an open door to draft content. Disable indexing on staging, and password-protect anything sensitive.
Assuming robots.txt is enforced. Robots.txt is voluntary. Treat it as your published policy, and use password protection or an edge layer for enforcement where it matters.
Leaving the global noindex toggle on after launch. This one hides your whole site from search engines. After every publish, spot-check that your key pages are still indexed in Search Console.
For the full technical background on robots.txt syntax, see Google's robots.txt specification. For a Webflow-specific walkthrough of the settings, the Finsweet Webflow SEO guide to robots.txt covers the same ground with screenshots.
FAQ
Can I edit robots.txt in Webflow?
Yes. Go to Project Settings, open the SEO tab, and add your rules in the robots.txt field under Indexing. Save and publish your site for the changes to go live. The custom robots.txt field is available on paid site plans with a connected custom domain.
Does Webflow have a plugin to block AI crawlers?
No. Webflow has no plugin system like WordPress. You block AI crawlers with the built-in robots.txt field, page-level indexing settings, and password protection. There is no app to install.
Will blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI's training crawler and has no role in Google Search. Googlebot is a separate crawler. As long as your robots.txt allows Googlebot, blocking GPTBot has no effect on your Google rankings.
Should I use noindex or robots.txt to keep a page away from AI crawlers?
Use robots.txt to stop crawlers from fetching a page. Use noindex to keep a page out of search indexes while still letting crawlers read it. Never put both on the same URL, because a robots.txt block prevents the crawler from ever seeing the noindex tag.
How do I block AI crawlers on my Webflow staging site?
Turn on the setting to disable search engine indexing for the staging subdomain, and password-protect anything sensitive. Staging sites are a common leak for draft content, so protect them before launch, not after.
How long does it take for a robots.txt change to take effect on Webflow?
The file itself updates as soon as you publish the site. Crawlers may cache the old file for up to a day, and some bots only re-check it periodically, so allow a week before judging whether a blocked bot has actually stopped visiting.
