Docusaurus sites are static, so blocking AI crawlers is simpler than on most platforms: you add one plain text file, static/robots.txt, rebuild, and deploy. No plugin, no server config, no dashboard toggle. The catch is placement. If your docs live under a subpath such as org.github.io/project/, a robots.txt filed at that subpath is ignored by crawlers, because they only read the file at the root of the host. Get the location right and the rest is a fifteen-minute job.
In this guide you will learn where the file goes in a Docusaurus project, which AI crawlers to block and which to keep, a copy-ready ruleset that blocks training bots while keeping your docs visible in ChatGPT search and Google AI Overviews, and how to test the result before you ship it.
Why Docusaurus Sites Need Their Own Approach
Docusaurus builds your documentation into plain HTML files. Search engines and AI crawlers can read every page without running JavaScript, which is great for visibility — and it also means your content is unusually easy to collect. A crawler does not need to render anything or fight a JavaScript framework. It requests a page, gets the full text, and moves on.
That matters because documentation is exactly the kind of content AI systems want. API references, tutorials, and how-to guides are clean, structured, and written in plain language. If you publish docs for a commercial product, a training crawler can absorb your entire knowledge base in one pass. Blocking the trainers while staying visible in AI search is a sensible middle ground, and Docusaurus gives you a clean way to do it.
Unlike WordPress, there is no plugin layer to lean on. Docusaurus does not generate a robots.txt file for you, and there is no admin panel where you paste rules. You own the file directly, which is actually an advantage: what you write is exactly what gets served, with no plugin rewriting your rules or caching an old version.
Where Robots.txt Goes in a Docusaurus Project
Create the file at static/robots.txt, in the project root next to docusaurus.config.js. Everything under static/ is copied unchanged into the root of the build output, so after npm run build the file lands at build/robots.txt and is served at https://yourdomain.com/robots.txt. It needs no front matter, no extension change, and no registration anywhere in your config.
If your project uses the staticDirectories option, the same rule applies: any directory listed there is copied to the output root. A setting like staticDirectories: ['public', 'static'] lets you keep the file in public/ instead, but the default static/ folder is the simplest choice and the one most projects already have.
Verify the placement after building. Run the build locally and check that build/robots.txt exists. Then open https://yourdomain.com/robots.txt in a browser after deploying and confirm you see your rules. If the URL returns a 404, the file is in the wrong folder or your host is serving a stale build.
The baseUrl Trap: When Your Rules Carry No Force
This is the mistake that catches most Docusaurus sites, and it is worth understanding before you write a single rule. Crawlers read robots.txt only at the root of the host — https://yourdomain.com/robots.txt — and nowhere else. Docusaurus, however, lets you serve a site from a subpath using the baseUrl setting. A project site on GitHub Pages, for example, often lives at org.github.io/project/ with baseUrl: '/project/'.
In that setup, your static/robots.txt is copied to build/robots.txt but served at /project/robots.txt. No crawler looks there. Your rules exist, they are correct, and they do absolutely nothing. The file at the host root — org.github.io/robots.txt — belongs to the account, not to your repository, and you cannot edit it from your project.
There are three honest fixes. The cleanest is to serve your docs from a custom domain or a subdomain, so the site root is yours and baseUrl is /. The second is to put the site at the root of a dedicated host, such as a Netlify or Vercel deployment with its own domain. The third, if you must stay on a project subpath, is to accept that robots.txt cannot protect you there and use other layers instead: a Cloudflare rule in front of the site, or meta tags on individual pages. Do not waste time polishing a robots.txt file that no crawler will ever read.
Before you go further, check your own setup. If your site's baseUrl is / and you have a custom domain, you are fine. If it is anything else, fix the hosting shape first.
Which AI Crawlers to Block and Which to Keep
Not all AI crawlers do the same job, and blocking the wrong ones costs you visibility. There are three groups, and the split matters more than the block itself.
Training crawlers collect content to train models. GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google's opt-out token for Gemini training), CCBot (Common Crawl), and Meta's training crawlers fall in this group. Blocking them means your docs are not used to train future models. It does not remove you from search results.
AI search crawlers build the index that answer engines cite. OAI-SearchBot powers ChatGPT search citations, Claude-SearchBot feeds Claude's answers, and PerplexityBot feeds Perplexity. Blocking these removes your docs from AI answers entirely. If you want developers to find your docs through ChatGPT or Perplexity, keep these allowed.
User-triggered fetchers visit a page only when a specific user asks a chatbot to fetch it — ChatGPT-User and similar agents. Robots.txt may not apply to these, according to the operators themselves, because the fetch happens on behalf of a user who pasted your link. You cannot reliably block them with robots.txt, and trying to do so is not worth the effort.
The recommended split for most documentation sites: block the trainers, allow the search crawlers. Your content stays citable in AI answers while staying out of training datasets.
The Copy-Ready Ruleset for Docusaurus
Here is a ruleset that implements that split. Paste it into static/robots.txt, replace the domain in the sitemap line with your own, and rebuild.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlA few notes on this file. First, the order of groups does not matter — crawlers only read the group that matches their own user-agent string, and the most specific match wins. Second, Google-Extended is a control token, not a separate crawler: blocking it opts your content out of Gemini training and grounding without touching Google Search, rankings, or AI Overviews, which all depend on Googlebot. Third, the User-agent: * group at the end keeps everything else, including Googlebot and Bingbot, fully allowed.
If you would rather block every AI crawler and disappear from AI answers as well, add Disallow groups for OAI-SearchBot, Claude-SearchBot, and PerplexityBot too. That is a legitimate choice for private or commercially sensitive documentation. Just make the decision deliberately, because it is easy to forget you made it and then wonder why ChatGPT never cites your docs.
Protecting Versioned Docs and Search Pages
Docusaurus sites have a few URL patterns worth thinking about separately. Versioned documentation lives under paths like /docs/1.0/ and /docs/2.0/. You rarely want a training crawler to spend its time on five old versions of the same page, but you also do not want to block old versions from search if users still run them. The practical answer is to leave versioned docs crawlable for search bots and let the AI-trainer blocks above handle the training side. The trainers are already blocked site-wide, so versioned pages need no extra rules.
The built-in search page, usually at /search/, is different. It is a JavaScript app with no useful content of its own, and crawlers that follow it waste requests. Add a targeted rule for it:
User-agent: *
Disallow: /search/Staging and preview deployments deserve a separate rule. If you deploy pull-request previews on Netlify or Vercel, each preview gets its own URL, and you do not want search engines or AI crawlers indexing half-finished docs. The simplest approach is a preview-only robots.txt that blocks everything — User-agent: * with Disallow: / — served only on preview domains. Many teams do this with a small build-time check that swaps the file based on the deployment context. If your host supports branch-based config, set it there rather than in the Docusaurus project itself, so production is never accidentally blocked.
Testing Before You Deploy
Robots.txt mistakes are silent. A wrong rule does not throw an error; it just quietly blocks the wrong bot or, worse, blocks Googlebot from your docs. Test before every deploy, not after.
Start with the free robots.txt tester: paste your file and a user-agent such as GPTBot or Googlebot, and check which paths each one can reach. Confirm that GPTBot is blocked everywhere, that OAI-SearchBot can reach your docs, and that Googlebot can reach everything. Then run the same check with the robots.txt generator if you want a clean file built from your choices instead of hand-editing text.
After deploying, verify the live file. Open https://yourdomain.com/robots.txt in a browser and confirm the content matches what you wrote. Then check Google Search Console's robots.txt report, which shows the file exactly as Google fetched it and flags syntax errors. Remember that Google caches robots.txt for up to a day, and OpenAI and Meta say their changes can take about 24 hours to register, so do not expect instant results.
Finally, watch your server logs or your host's analytics for a few days after the change. You should see GPTBot and ClaudeBot requests drop to zero while OAI-SearchBot and Googlebot continue normally. If GPTBot keeps arriving, check two things: that the file is really at the host root, and that your baseUrl is /. A persistent GPTBot usually means the file is filed under a subpath and nobody is reading it.
What Robots.txt Cannot Do on Docusaurus
Robots.txt is a request, not a lock. Well-behaved crawlers from OpenAI, Anthropic, Google, and Meta honour it. Scrapers and poorly behaved bots ignore it entirely. If your docs contain genuinely sensitive material — unreleased API keys in examples, internal architecture notes, customer data — do not rely on robots.txt. Remove the content or put it behind authentication.
Blocking a page in robots.txt also does not remove it from search results. A blocked page can still appear in Google as a bare URL if other sites link to it, because Google never fetches the page to see any noindex tag on it. If you want a page out of search results, keep it crawlable and add a noindex tag instead. In Docusaurus, that means adding the page to your sitemap exclusion or setting front matter that marks it as not indexed, depending on how your site is configured.
And remember the timing limits. Blocking a training crawler today affects what it collects from now on. It does not remove data already gathered, and it does not un-train a model. The value is forward-looking: your next release, your next guide, your next API reference stays out of the training set.
Keeping the Rules Current
Bot names change. Anthropic's old anthropic-ai and Claude-Web tokens are no longer used, so rules naming only them do nothing today. New crawlers appear every few months — DeepSeekBot, ByteSpider, and others arrived with little warning. Review your robots.txt at least once a quarter, check the current user-agent names against the operators' own documentation, and update the file the same way you update your docs: a commit, a build, a deploy.
It is also worth re-checking your hosting shape whenever you change it. Moving from GitHub Pages to a custom domain, changing baseUrl, or adding a new deployment target can all silently move your robots.txt out of the host root. A two-minute check of the live URL after any hosting change saves weeks of wondering why the rules stopped working.
FAQ
Where does robots.txt go in a Docusaurus site?
At static/robots.txt in the project root. Docusaurus copies everything in static/ unchanged into the build output, so the file is served at your domain root after a build and deploy. No plugin or config change is needed.
Does robots.txt work on GitHub Pages project sites?
Not reliably. If your site lives at a subpath like org.github.io/project/, crawlers only read org.github.io/robots.txt, which belongs to the account, not your repository. Use a custom domain or a subdomain so the site root is yours.
Will blocking GPTBot remove my docs from ChatGPT answers?
No. GPTBot collects training data; OAI-SearchBot powers ChatGPT search citations. They are separate crawlers. Blocking GPTBot while allowing OAI-SearchBot keeps your docs citable in ChatGPT while keeping them out of training data.
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether your content is used for Gemini training and grounding. Google Search, rankings, and AI Overviews depend on Googlebot, which is unaffected.
How long does it take for a robots.txt change to work?
Expect about 24 hours. Google caches robots.txt for up to a day, and OpenAI and Meta report similar delays. The change is not retroactive: it stops future collection, it does not remove data already gathered.
Do I need a plugin to manage robots.txt on Docusaurus?
No. A static file is the simplest and most reliable method. Plugins add a layer that can cache or rewrite your rules. Edit static/robots.txt directly, rebuild, and deploy.
For more on the wider picture, see our guide to blocking AI crawlers with robots.txt and the complete list of AI crawlers for 2026. If you run docs on another static host, the same split — block trainers, allow search bots — applies, and our free robots.txt generator will build the file for you.
