Articles about "baseUrl robots.txt"

Docusaurus makes blocking AI crawlers easy — one file in static/robots.txt does the job. But the baseUrl trap catches most docs sites: rules filed under a project subpath carry no force. This guide shows the exact setup, which bots to block, and how to verify it.