Yes, you can block AI crawlers on a Ruby on Rails site with a single file. Create public/robots.txt, add a Disallow: / group for each training crawler you want to keep out — GPTBot, ClaudeBot, Google-Extended, and CCBot — and deploy. Rails serves anything in public/ as a static file, so the file shows up at yourdomain.com/robots.txt with no extra code. If you need different rules for staging and production, swap the static file for a small controller route instead. This guide walks through both approaches, which bots to block and which to leave alone, and how to check that it actually works.
Before you touch anything, one honest warning: robots.txt is a request, not a lock. Google itself says that well-behaved crawlers follow it, but nothing forces a bot to obey. It stops the crawlers run by OpenAI, Anthropic, Google, and Common Crawl, because those companies publish their user agents and promise to respect the file. It does nothing against a scraper that ignores the rules. Later in this guide you will see what to do about those bots too.
How robots.txt works on a Rails app
A crawler that visits your site asks for one file first: /robots.txt on your domain. That file is plain text. It lists user agents — the names crawlers use to identify themselves — and says which paths each one may or may not request. The format is an internet standard, RFC 9309, and every major search engine and AI company follows it.
Rails has a built-in way to serve this file. Anything you put in the public/ folder of your app is served as a static file by the ActionDispatch::Static middleware. A file at public/robots.txt is answered at /robots.txt before your routes are even checked. No controller, no route, no gem.
There is one setting to know about. In production, Rails only serves files from public/ itself when config.public_file_server.enabled is true. The default in config/environments/production.rb ties this to an environment variable:
config.public_file_server.enabled = ENV["RAILS_SERVE_STATIC_FILES"].present?On many hosts this is already handled for you — Heroku's Ruby buildpack sets RAILS_SERVE_STATIC_FILES for you, and the Dockerfiles that Rails generates set it too. If your host puts nginx or another web server in front of Rails, that server reads public/ directly, which also works. The only setup where /robots.txt returns a 404 is one where neither Rails nor the web server is serving static files, so if you see a 404, check this setting first.
Step 1: The static file in public/robots.txt
For most Rails apps this is the whole job. Create the file public/robots.txt in your project, paste in the rules below, commit, and deploy.
# Allow search engines, block AI training crawlers
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /users/
Disallow: /api/
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlRead it from the top. The first group applies to every crawler (User-agent: *) and lets it crawl the whole site except your admin area, user pages, and API endpoints. Change those three paths to match your own app — a Rails app with Devise, for example, might also want to block /users/sign_in and /users/password, and an app with Active Admin would block /admin/ as shown.
Then come four small groups, one per training crawler, each saying Disallow: / — stay out of the whole site. The key point is that a crawler follows only the most specific group that names it. GPTBot matches its own group, not the * group, so it is blocked everywhere even though the general group allows everything. Googlebot, which has no group of its own here, follows the * group and keeps crawling normally. That is how you block AI training without touching your Google rankings.
Replace example.com in the sitemap line with your own domain. If you do not have a sitemap yet, leave the line out — it is optional, but a sitemap helps search engines find your pages, so it is worth adding when you have one.
Which AI crawlers to block — and which to leave alone
Not every bot with an AI company's name on it does the same job. The companies now split their crawlers into three kinds, and the choice matters.
- Training crawlers collect pages that may be used to train future models. GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), and CCBot (Common Crawl) are the ones most Rails site owners block. Blocking them does not affect search.
- Search crawlers build the index that an AI assistant searches when it answers a question with sources. OAI-SearchBot (OpenAI) and Claude-SearchBot (Anthropic) are in this group. Block them and your pages stop appearing as sources in ChatGPT and Claude search answers.
- User-triggered fetchers open a page because a real person asked the assistant to look at it. ChatGPT-User and Claude-User do this. OpenAI says robots.txt rules may not apply to ChatGPT-User, because the request comes from a user, not from a crawl.
The default recommendation for most Rails apps is simple: block the four training crawlers and leave the search crawlers alone. You stay out of training data, and you keep the chance of being cited in AI answers. Google-Extended deserves a special note: it controls whether Google uses your content to improve its Gemini models. Blocking it has no effect on Google Search or on AI Overviews, which are driven by Googlebot, so it is safe to block.
If your content is your product — paid courses, a subscription newsletter, a marketplace with unique listings — you may want to block the search crawlers too. That is a business decision, not a technical one. Just know the trade: blocking OAI-SearchBot means your pages will not show up in ChatGPT search answers, though they can still appear as plain navigation links.
Step 2: A dynamic robots.txt when one file is not enough
The static file breaks down in one common situation: you run the same code in more than one environment. Your staging app on a temporary domain is a copy of production, so it serves the same permissive robots.txt — and search engines can index your staging site, which is exactly what you do not want.
The fix is to serve robots.txt from a controller instead of a file, and let the code decide which rules to return. First, delete public/robots.txt. This step is easy to forget and it matters: a static file in public/ is answered before your routes, so if it is still there, your controller will never see the request.
Then add a route. In config/routes.rb:
# Dynamic robots.txt — must come before other routes
get "robots.txt", to: "robots#index", format: :textNext, a small controller at app/controllers/robots_controller.rb:
class RobotsController < ApplicationController
def index
if Rails.env.production?
render plain: production_rules, content_type: "text/plain"
else
render plain: "User-agent: *\nDisallow: /\n", content_type: "text/plain"
end
end
private
def production_rules
<<~TXT
User-agent: *
Allow: /
Disallow: /admin/
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
TXT
end
endProduction gets the normal rules. Every other environment — development, staging, review apps — gets Disallow: / for all crawlers. You can make the check smarter if you need to: compare request.host against your production domain instead of checking the environment, which handles the case where staging also runs with RAILS_ENV=production. Either way, the rule is the same: the file the crawler sees should depend on where the app is running, not on which file you remembered to deploy.
One more detail: keep this controller light. Skip authentication filters and any before_action that redirects logged-out users, or crawlers will get a redirect instead of your rules. If your ApplicationController has a global before_action for login, add skip_before_action for it in the robots controller.
The Rack middleware option
If you would rather keep robots.txt out of your controllers entirely, a tiny Rack app does the same job. Rack sits between the web server and Rails, so it can answer /robots.txt before the request reaches the router at all.
# lib/robots_middleware.rb
class RobotsMiddleware
def initialize(app)
@app = app
end
def call(env)
if env["PATH_INFO"] == "/robots.txt"
body = Rails.env.production? ? production_rules : "User-agent: *\nDisallow: /\n"
[200, { "Content-Type" => "text/plain" }, [body]]
else
@app.call(env)
end
end
private
def production_rules
File.read(Rails.root.join("config", "robots.txt"))
end
endWire it up in config/application.rb:
config.middleware.use RobotsMiddlewareWith this approach your production rules live in config/robots.txt, a plain text file you can edit without touching Ruby. The controller version is easier to test with request specs, the middleware version keeps the logic out of your controllers. Either one works; pick the one your team will actually maintain.
Blocking bots inside the app: before_action and Rack::Attack
Sometimes robots.txt is not the right tool. A crawler that ignores robots.txt will keep hitting your pages, and a file that says "please stay out" will not slow it down. For those cases you can act inside the app.
A before_action in ApplicationController can turn away requests by user agent:
class ApplicationController < ActionController::Base
BLOCKED_BOTS = %w[GPTBot ClaudeBot CCBot].freeze
before_action :block_unwanted_bots
private
def block_unwanted_bots
agent = request.user_agent.to_s
if BLOCKED_BOTS.any? { |bot| agent.include?(bot) }
head :forbidden
end
end
endThis returns a 403 for any request whose user agent contains one of those names. It works, but it has limits. It runs on every request, including requests from real users, so keep the list short and the check cheap. It is also easy to fool: a bot can change its user agent string in seconds. Treat it as a backstop for bots that identify themselves honestly but ignore robots.txt, not as a security boundary.
For real protection against aggressive crawlers, rate limiting is the better tool. The rack-attack gem lets you throttle requests by IP address, which catches bots no matter what user agent they claim. A common setup is to allow a reasonable number of requests per minute per IP and return a 429 response past that limit. Rate limiting protects your server from being hammered; robots.txt protects your content from being used. They solve different problems, and a site with valuable content usually wants both.
Verifying it works
Do not trust that the file is live just because you deployed it. Check it the way a crawler would.
First, fetch it from the command line:
curl https://example.com/robots.txtYou should see your rules as plain text, with a 200 status. Check the headers too — curl -I https://example.com/robots.txt should show Content-Type: text/plain. If you get HTML instead, something is wrong: usually the static file is missing and a catch-all route is rendering your homepage, or a controller is redirecting the request.
Second, test the rules against real paths. The robots.txt tester lets you paste your file and check whether a given user agent is allowed on a given path. Try GPTBot against your homepage (it should be blocked) and Googlebot against the same page (it should be allowed). If you use the dynamic controller version, test staging separately and confirm it returns Disallow: / for everything.
Third, watch your logs. Rails logs every request, including the user agent, so a quick search of log/production.log for GPTBot tells you whether the bot is still visiting and which pages it requests. On Heroku, use heroku logs --tail and filter for the bot name. A blocked bot will still show up in your logs — it still requests pages — but after a few days you should see its requests drop off, because well-behaved crawlers read robots.txt first and stop asking for disallowed paths. If requests keep coming at the same rate weeks later, that bot is ignoring your file, and it is time for rate limiting or a firewall rule.
One timing note: Google caches robots.txt for up to 24 hours, and other crawlers cache it too. Changes are not instant. If you fix a mistake, expect a day before every crawler sees the new file. You can speed up Google's side by asking for a re-crawl in Google Search Console, but there is no button that forces every bot to re-read the file at once.
Deployment notes: Heroku, Render, Fly.io, and Kamal
The static-file approach works on every common Rails host, because public/robots.txt is part of your code and ships with every deploy. The details differ slightly.
Heroku. The Ruby buildpack sets RAILS_SERVE_STATIC_FILES, so Rails serves public/ itself and your robots.txt just works. Nothing to configure. The dynamic controller version works here too, since it is plain Rails code.
Render. Set the RAILS_SERVE_STATIC_FILES environment variable to true in your web service settings if you serve static files through Rails. If you use Render's static site or CDN setup in front of the app, make sure /robots.txt is not cached for long — a stale cached copy will keep serving old rules after you deploy a fix.
Fly.io. The Dockerfile that Rails generates for Fly sets the static-files variable, so the static file is served by Rails inside the container. Your fly.toml needs no special handling for robots.txt.
Kamal. Kamal deploys your app in a Docker container, and Rails 8 ships with Thruster, a small proxy inside the container that serves static files and compresses responses. Your robots.txt is served from public/ like any other static file. With Kamal, the dynamic controller version is often the better choice anyway, because Kamal makes it easy to spin up review apps and staging targets — exactly the situation where per-environment rules earn their keep.
Whichever host you use, the check after every deploy is the same: run curl against the live URL and confirm the rules you expect. Add it to your deploy checklist alongside checking that the homepage loads.
When robots.txt is not enough
Some bots ignore robots.txt entirely. They scrape content, hammer your server, and burn through your database connections. For those, you need enforcement, not requests.
Rate limiting. The rack-attack gem throttles by IP, which works regardless of user agent. Start generous — real users behind a shared office or university IP should never hit the limit — and tighten only if a specific IP is clearly abusive. A 429 response is the standard signal that a client should slow down, and well-behaved tools respect it.
Cloudflare. If your site sits behind Cloudflare, its bot management can block known AI crawlers at the edge, before they ever reach your Rails app. Cloudflare also lets you write firewall rules that match user agents, so you can block a bot by name without touching your code. This is the strongest option for bots that ignore robots.txt, because the request never reaches your server at all. The trade is that you are now depending on Cloudflare's bot list, so keep your robots.txt in place as the base layer — it is the one every crawler can read, no matter which CDN you use.
Authentication for sensitive areas. Robots.txt never hides anything. The file is public — anyone can read it, and the paths you list in it are a map of the parts of your site you consider sensitive. If a path must not be seen at all, protect it with login, not with a Disallow line. Admin panels, user account pages, and internal tools should require authentication whether or not a crawler is blocked from them.
Common mistakes Rails developers make
- Leaving the static file in place when switching to dynamic rules.
public/robots.txtwins over any route. Delete it before you deploy the controller version, or your new rules will never be served. - Blocking everything in production by accident. The classic error is deploying a staging
Disallow: /file to production. Check the live file after every deploy. A single bad deploy can drop your pages out of search results, and recovery takes days. - Assuming robots.txt removes pages from search results. It does not. It tells crawlers not to fetch a page, but if other sites link to that page, Google may still list its URL without content. To remove a page from results, use a
noindexmeta tag or anX-Robots-Tagheader, not robots.txt. - Using Crawl-delay for Google. Google ignores
Crawl-delayentirely. If Googlebot is crawling too hard, use the crawl-rate settings in Google Search Console instead. - Forgetting the file is per-host. Robots.txt applies to one host only. Your staging domain, your API subdomain, and your main domain each need their own file. A rule on
example.comdoes nothing forapi.example.com. - Writing rules for bot names you have not verified. Bot names change. Check the crawler's own documentation before adding a token — a misspelled user agent matches nothing, and your "block" quietly does nothing.
If you want to draft the file before touching your app, the free robots.txt generator builds a valid file from checkboxes, and the htaccess generator covers the server-level rules for Apache setups. The same approach works on other frameworks too — see our guides on blocking AI crawlers on Django and blocking AI crawlers on Laravel, and our comparison of robots.txt vs htaccess vs Cloudflare for AI bot blocking.
Two official references are worth bookmarking: Google's robots.txt specification for the exact rules every crawler is expected to follow, and OpenAI's bot documentation, which lists GPTBot's user agent and the IP ranges it crawls from.
FAQ
Does blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI's training crawler and has nothing to do with Google Search. Googlebot is a separate crawler with its own user agent, and it follows the * group in your file, not the GPTBot group. Blocking GPTBot, ClaudeBot, Google-Extended, and CCBot leaves your Google rankings untouched. Only blocking Googlebot itself — or a careless Disallow: / under User-agent: * — would hurt search visibility.
Where do I put robots.txt in a Rails app?
Put it in public/robots.txt. Rails serves files from public/ as static files, so the file appears at yourdomain.com/robots.txt with no route or controller. Make sure config.public_file_server.enabled is true in production, which it is by default when RAILS_SERVE_STATIC_FILES is set. If you need different rules per environment, delete the static file and serve the rules from a controller route instead.
How do I block AI crawlers on staging but not production?
Serve robots.txt dynamically. Delete public/robots.txt, add a robots#index route, and return Disallow: / for every user agent when the app is not running in production. Check Rails.env.production? or compare request.host to your production domain. A Rack middleware that reads a different file per environment does the same job without a controller.
How long does it take for AI crawlers to respect a new robots.txt?
Usually within a day or two. Google caches robots.txt for up to 24 hours, and other crawlers cache it for similar periods, so a change is not instant. Well-behaved crawlers read the file before crawling, so they pick up the new rules on their next visit. If a bot keeps requesting disallowed pages after a week, it is ignoring the file and you need rate limiting or a firewall rule instead.
Can I block AI crawlers and still appear in ChatGPT search answers?
Yes, if you block only the training crawlers. GPTBot collects content that may be used for training, while OAI-SearchBot builds the index ChatGPT searches when it answers questions. OpenAI treats each setting independently, so you can disallow GPTBot and still allow OAI-SearchBot. The same split exists at Anthropic: ClaudeBot is for training, Claude-SearchBot is for search. Block the training bot, allow the search bot, and you keep the visibility while staying out of training data.
Should I use robots.txt or Cloudflare to block AI bots?
Use both. Robots.txt is the base layer: it is a standard every reputable crawler reads, it costs nothing, and it works no matter where your app is hosted. Cloudflare is the enforcement layer: it can block bots at the edge, including ones that ignore robots.txt, before they reach your server. Start with robots.txt, verify it in your logs, and add Cloudflare rules only for bots that keep coming back after being told to stay away.
Blocking AI crawlers on Rails comes down to one decision: static file or dynamic route. A static public/robots.txt covers most apps, takes five minutes, and deploys with your normal code. The dynamic version is worth the extra code the moment you run staging or review apps from the same codebase. Either way, block the training crawlers, keep the search crawlers if you want to be cited, verify the live file with curl, and watch your logs for bots that ignore the rules. Robots.txt is a polite request — make it correctly, and the crawlers that matter will listen.
