Robots.txt is a polite request. Cloudflare is enforcement. For sites that need harder protection, Cloudflare's free and paid tiers offer powerful bot management tools.
Bot Fight Mode (Free)
Enable Bot Fight Mode in Cloudflare → Security → Bots. This automatically challenges bots exhibiting machine-like behavior, including many undeclared AI scrapers.
Custom Rules for User-Agent Matching
http.user_agent contains "GPTBot" or http.user_agent contains "CCBot" or http.user_agent contains "Anthropic-ai"
Action: Block. This enforces your robots.txt at the network level regardless of bot compliance.
Rate Limiting
Set rate limits of 10 requests per minute for /blog/ and /articles/. Legitimate users rarely hit this. Scrapers always do.
