Yes, you can block AI crawlers on a GitHub Pages site, and the whole job is one text file: a robots.txt placed at the root of your repository. GitHub Pages is static hosting, so there is no server panel, no firewall rule, and no plugin to install. Robots.txt is the lever you have, and this guide shows how to use it properly, what it cannot do, and the workarounds for the gaps.
GitHub Pages serves whatever files you commit, exactly as they are. That simplicity is the attraction, and it is also the limit. You cannot run server-side code, you cannot inspect incoming requests, and you cannot block a bot by IP address. Everything you do to control crawlers has to happen through files the crawler reads voluntarily: robots.txt, and meta tags on individual pages.
Step 1: Add robots.txt to the right repository
The single most common mistake on GitHub Pages is putting robots.txt in the wrong place. Crawlers only read robots.txt at the root of a domain, so the file must end up served at your domain's root path, for example yoursite.com/robots.txt.
- User or organization site (username.github.io): put robots.txt in the root of the username.github.io repository. That one file governs the whole domain, including every project page served under it.
- Project site (username.github.io/my-project): a robots.txt inside the project repository is served at username.github.io/my-project/robots.txt, which crawlers ignore. Only the user site's root file counts. If you only have project sites and no user site, you cannot set domain-level robots.txt rules at all.
- Custom domain (docs.example.com): put robots.txt in the root of the repository that has the custom domain configured. That file is served at the custom domain's root.
- Docs folder publishing: if your site publishes from a docs folder on a branch, robots.txt must live inside that docs folder, not at the repository root, because only the docs folder is published.
For a Jekyll site, place robots.txt in your source root, next to your config file. Jekyll copies files that are not posts, layouts, or underscore-prefixed files straight into the built site, so a plain robots.txt in the source root lands at the published root. If you use a different static generator, the equivalent is your public or output folder's root: whatever ends up deployed as the site root.
Step 2: Write rules that block trainers but keep search working
The goal for most site owners is not to block everything. It is to stop AI training crawlers while keeping Google and other search engines able to index the site, and keeping answer-engine crawlers able to fetch pages you want cited. That means separate rules for separate bots, not one blanket block.
A practical starting file looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /Read it from the top. GPTBot is OpenAI's training crawler, ClaudeBot is Anthropic's training crawler, CCBot is Common Crawl's bot, and PerplexityBot is Perplexity's crawler. Google-Extended is the token that opts your content out of Google's AI training use. The final rule allows everything else, which keeps Googlebot, Bingbot, and the answer-engine fetchers able to reach your pages.
One split matters more than the rest. Blocking GPTBot does not have to mean blocking ChatGPT citations, because ChatGPT's search feature fetches pages with a different crawler, OAI-SearchBot. If you want citations but not training, block GPTBot and leave OAI-SearchBot allowed. The same pattern applies at Anthropic: ClaudeBot trains models, while separate search and user fetchers retrieve pages for answers. Keeping those two ideas apart is the difference between a site that disappears from AI answers and a site that stays visible without feeding training data.
If writing these rules by hand feels error-prone, generate them instead: CrawlGuards' free robots.txt generator builds the file with AI-bot presets, and you paste the result into your repository.
Step 3: Verify the file is actually live
Commit and push, then wait a minute or two for GitHub Pages to rebuild. Then check three things:
- Open the URL directly. Visit yoursite.com/robots.txt in a browser. You should see exactly the file you committed, with no theme layout wrapped around it. If you see a 404 page styled like your site, the file is in the wrong folder.
- Test it against a bot. Paste the live URL into a robots.txt tester and check how it answers for GPTBot, ClaudeBot, and Googlebot. A rule that looks right in your editor can still be wrong if a typo merged two stanzas.
- Check for caching. GitHub Pages sits behind a CDN, so an old robots.txt can linger for a short while after a push. If the live file looks stale, wait a few minutes and hard-refresh before assuming the deploy failed.
What GitHub Pages cannot do about crawlers
No server-side blocking
Robots.txt is a request, not a lock. Well-behaved crawlers obey it, and the major AI companies' documented crawlers do. A scraper that ignores robots.txt will still reach your pages, and GitHub Pages gives you no way to stop it: no IP blocking, no user-agent filtering, no rate limiting, and no access logs to even see that it happened. If a specific bad actor is hammering your site, robots.txt alone will not fix it.
No per-page server headers
On a normal server you can send an X-Robots-Tag header to noindex a whole folder of files, including PDFs and images. GitHub Pages does not let you set custom headers, so for individual pages you fall back to a meta robots tag in the page itself. That works for HTML pages but not for files like PDFs, which have no head section to put a tag in. If a file must not be indexed, the practical answer on GitHub Pages is to not publish it there.
Your repository is public by default
This is the gap people miss. Blocking crawlers on your Pages site does nothing about the source repository on github.com, where the same content is browsable file by file. Search engines index public repositories too, and anyone can read your markdown source directly. If content is genuinely sensitive, it should not be in a public repository at all, regardless of what your robots.txt says. A private repository still publishes a public site, so privacy and crawler-blocking are two different problems with two different answers.
The Cloudflare workaround for real enforcement
If you have a custom domain, you can put Cloudflare in front of GitHub Pages and get the enforcement GitHub Pages cannot provide. Point your domain's DNS at Cloudflare, set the DNS record for your Pages site to proxied, and Cloudflare's firewall rules can block or challenge user agents before they ever reach GitHub's servers. Cloudflare also offers a managed option to block AI bots, which covers the known training crawlers without you maintaining a list.
This is a bigger setup than a robots.txt file, and it only works with a custom domain, not with a username.github.io address. But it is the standard answer for GitHub Pages owners who need more than a polite request: the request stays in robots.txt for well-behaved bots, and the firewall handles the rest. Our companion guide to blocking AI bots with Cloudflare walks through that setup.
Per-page control with meta robots tags
Sometimes you want most of the site open but one section closed, for example a draft docs section or an old version of a tutorial. Add a meta robots tag to those pages' head sections with a noindex value, and search engines will drop them from results while the rest of the site stays indexed. In Jekyll, this usually means adding a conditional to your default layout that outputs the tag when a page sets a front-matter flag. Keep the robots.txt permissive for those pages, because a page blocked in robots.txt can never be crawled to see its noindex tag, and it may stay indexed as a bare URL instead.
Common deployment mistakes that silently break your rules
A robots.txt file that never reaches the published root is the same as no robots.txt at all, and GitHub Pages has several quiet ways to produce that outcome. The first is the underscore rule: Jekyll ignores files and folders that start with an underscore, so a robots.txt tucked inside an underscore-prefixed folder will not be published. The file itself does not start with an underscore, but its parent folder might, and that is enough to lose it.
The second is the branch mismatch. GitHub Pages publishes from exactly one source: a branch, optionally a folder on that branch. If you commit robots.txt to your main branch but your site publishes from a gh-pages branch or a docs folder, the published site never sees the file. Always confirm which source is configured in the repository's Pages settings before debugging anything else.
The third is the theme default. Some Jekyll themes ship their own robots.txt or generate one during the build, and a theme-generated file can overwrite yours. After your first deploy with a custom file, open the live URL and read it. If the content is not what you wrote, search your theme and plugins for a robots.txt template and remove or override it.
FAQ
Does GitHub Pages support a robots.txt file?
Yes. It is just a static file like any other. Commit it to the published root of your site and it is served at your domain's robots.txt path within minutes.
Can I block AI crawlers on a project page?
Not with that project's own robots.txt, because crawlers only read the domain root. You need control of the user site repository or a custom domain to set domain-level rules. On a project page, per-page meta robots tags are your only lever.
Will blocking GPTBot hurt my Google rankings?
No. GPTBot and Googlebot are different crawlers with different jobs. Blocking GPTBot does not affect Google Search indexing, provided you leave Googlebot allowed. See our guide on whether blocking GPTBot hurts SEO for the details.
How long until crawlers respect a new robots.txt?
Major crawlers re-fetch robots.txt regularly, typically within a day, and often within hours for active sites. Pages already crawled and copied before the change are not retroactively deleted by a robots.txt edit; the file controls future crawling, not past copies.
Should I also add an llms.txt file?
It costs nothing to add one alongside robots.txt, and it declares your site's key pages in a format built for AI tools. Treat it as a small bonus, not a replacement: robots.txt is the file crawlers actually check, and llms.txt is rarely requested by comparison. Our llms.txt generator can build one for you.
What if I use GitHub Actions to deploy instead of Jekyll?
The same rule applies: whatever file ends up at the root of the deployed output is what crawlers read. Put robots.txt in your build's source static folder so the build copies it to the output root, and verify the live URL after each deploy.
