AI Search
Which Industries Block AI Crawlers? News Sites Lead by a Wide Margin (2026 Robots.txt Data)
News sites block AI crawlers more than any other industry: 41 of 69 news robots.txt files (59%) disallow GPTBot, vs 2 of 97 in finance. Fresh Oct 2026 robots.txt data: news leads at 59%, finance barely blocks. See the numbers and what to do.
Rahul Sharma
11 min
