To opt out of AI training beyond robots.txt, stack several signals: a TDM reservation that states your rights in machine-readable form, an ai.txt file that declares your AI-use policy, clear license and terms-of-use language, and technical barriers such as login walls or firewall rules for crawlers that ignore polite requests. Robots.txt is only one layer, and it is the weakest one on its own, because it is a voluntary request with no legal force in most places. No single method guarantees that your content will never be used for training. A layered opt-out is the strongest position available to a site owner today, and this guide shows how to build it.
The confusion around AI opt-outs comes from treating every signal as if it were a lock. It is not. Robots.txt, TDM reservations, ai.txt files, and licenses are all declarations of preference. Some AI companies honor them carefully. Some ignore them. Some have been caught training on data they were told not to touch. That does not make declarations pointless — they create a paper trail, they work against the companies that do comply, and in the EU they carry real legal weight. But they work best as a stack, not as a single file you publish and forget.
Why robots.txt alone is not enough
Robots.txt is the oldest and most widely supported crawler-control mechanism on the web, and it is still the right starting point. A bot-specific rule such as User-agent: GPTBot followed by Disallow: / tells OpenAI's training crawler to stay out. Major AI companies have documented their crawler user-agents and say they honor these rules.
But robots.txt has three hard limits. First, it is voluntary. Nothing forces a crawler to read it or obey it. Scrapers, data brokers, and badly behaved bots routinely ignore it. Second, it only governs crawling, not use. A bot that respects your robots.txt will not fetch your pages — but content it already collected, or collected from somewhere else that republished your work, is outside its reach. Third, in most jurisdictions robots.txt has no settled legal status as a rights reservation. It is a technical signal, not a legal instrument. Courts have treated it in different ways, and nobody should assume it settles a copyright question on its own.
That is why the rest of this guide exists. Each additional layer covers a weakness of the one before it: legal reservations cover the legal gap, policy files cover the machine-readability gap, and technical barriers cover the enforcement gap.
Layer 1: Get robots.txt right first
Before adding anything exotic, make sure your foundation is solid. Block the training crawlers you do not want, by name, and leave the crawlers you do want alone. At minimum, most publishers who opt out block GPTBot (OpenAI training), ClaudeBot (Anthropic training), CCBot (Common Crawl, whose dataset feeds many models), and Bytespider (ByteDance). Keep Googlebot, Bingbot, and the answer-engine crawlers you want traffic from — OAI-SearchBot, Claude-Web, PerplexityBot — allowed unless you have decided to block them too.
Two details matter. First, spell the user-agent names exactly right. A misspelled token silently fails open: the bot will not match your rule and will crawl freely. Second, test the file after every change with a robots.txt tester, and generate clean rules with a robots.txt generator rather than hand-writing them from memory. Robots.txt is necessary. It is just not sufficient.
Layer 2: The TDM Reservation Protocol (the legal layer)
The strongest formal opt-out available today is the TDM Reservation Protocol, developed by the W3C community. It implements the opt-out mechanism in Article 4 of the EU Copyright in the Digital Single Market Directive (Directive 2019/790). Article 4 says that text and data mining is permitted unless the rights holder has expressly reserved their rights "in an appropriate manner, such as machine-readable means." The TDM Reservation Protocol is exactly that machine-readable means.
It works through two properties: tdm-reservation, which is 1 when rights are reserved and 0 when they are not, and tdm-policy, which points to a URL where your full policy lives. You can express those properties in four ways, in priority order: a site-wide file, HTTP headers, HTML meta tags, or document metadata.
Option A: The site-wide file. Publish a JSON file at /.well-known/tdmrep.json:
{
"@context": "http://www.w3.org/ns/tdmrep",
"tdm-reservation": 1,
"tdm-policy": "https://yoursite.com/tdm-policy"
}This is the simplest option for most sites: one file, whole-site coverage, easy to verify. Your tdm-policy page should state, in plain language, that text and data mining of your content — including for AI model training — is not permitted without a license, and how to contact you to request one.
Option B: HTTP headers. Send a tdm-reservation: 1 header with your responses. This is useful when different parts of your site have different policies — for example, reserving rights on your articles while allowing mining on your public documentation. It requires server configuration, so it suits sites with technical help.
Option C: HTML meta tags. Add <meta name="tdm-reservation" content="1"> to a page's head, plus a tdm-policy meta tag pointing to your policy URL. This is the per-page option, and it works in any CMS that lets you edit page headers, including WordPress, Ghost, and static sites.
Option D: Document metadata. For PDFs and EPUBs, the reservation can live in the document's own metadata. Relevant if you publish ebooks, whitepapers, or research PDFs.
Be honest about what this buys you. The TDM protocol is a real, standards-track mechanism with genuine legal significance in the EU, and the EU AI Act confirms that the copyright exceptions apply to general-purpose AI models. But adoption by AI companies is still uneven, and outside the EU its legal force is untested. Publish it because it is the correct formal reservation, not because it is a magic shield. It also pairs naturally with a public statement on your site: a short "AI use policy" page that humans can read, linked from your footer next to your copyright notice.
Layer 3: ai.txt — a plain-language AI policy file
ai.txt is a newer, simpler idea: a plain-text file at /ai.txt that states, in human- and machine-readable form, how your content may be used by AI systems. It was introduced by the team behind Spawning, and it is meant to complement robots.txt rather than replace it. A detailed legal overview of the TDM reservation options is available from JBB Rechtsanwälte. Where robots.txt speaks in crawl directives, ai.txt speaks in usage policy: which AI uses are allowed, which require permission, and where to ask.
An ai.txt file might say that your content may be quoted with attribution in AI answers, but may not be used to train models or reproduced wholesale. You can generate one with our free ai.txt generator, which walks through the choices and produces a file you can publish at your site root.
The honest caveat: ai.txt has limited adoption. Few AI companies have publicly committed to honoring it, and it has no legal standing anywhere. Its value is as an additional, unambiguous declaration — one more place where your position is on the record in a format machines can parse. Publish it because it costs ten minutes and adds a layer, not because it ends the discussion. And never treat ai.txt as a substitute for robots.txt or a TDM reservation. They answer different questions: robots.txt governs crawling, ai.txt governs use, and the TDM reservation governs legal rights.
Layer 4: Licenses and terms of use (the human layer)
Machine-readable signals only matter if a human or a court can also read your position. Your terms of use should state, in plain language, that automated collection of your content for AI training or model development is not permitted without a written license, and that your robots.txt and TDM reservation are part of that position. Put a short version in your footer — one sentence linking to the full policy is enough — and the full version on a dedicated page.
Licenses matter too. If you publish under a Creative Commons license, pick one that matches your intent. A CC BY-NC license permits reuse with attribution for non-commercial purposes only; a CC BY-ND license forbids derivatives. Neither was designed specifically for AI training, and their effect on model training is legally unsettled, but choosing the license that matches your intent is better than leaving the question unanswered. If you offer paid licensing for AI use, say so on your policy page, with a contact address. A reservation that offers a door is more likely to be respected than one that is only a wall.
One more human-layer step: make sure your copyright notice is current and visible. It will not stop a determined scraper, but it removes any argument that your position was unclear.
Layer 5: Technical barriers (the enforcement layer)
Declarations only work on crawlers that read them. For the rest, you need enforcement. This is where the stack stops asking and starts blocking.
- Firewall and bot-management rules. If your site runs behind Cloudflare or a similar service, use its bot controls to block known AI crawler user-agents at the edge. A firewall rule is enforced; robots.txt is requested. This is the right escalation for any crawler you have seen ignoring your robots.txt in your server logs.
- Rate limiting. Aggressive scrapers often fetch pages far faster than any polite crawler. Rate-limiting rules that slow down or block high-frequency request patterns catch scrapers that rotate user-agents, because the rule depends on behavior, not on the user-agent string being honest.
- Login walls and paywalls. Content behind authentication is not fetched by ordinary crawlers. If a specific section of your site is your most valuable original work — a research archive, a course, a dataset — gating it is the most reliable protection available. It has a cost: gated content is also invisible to search engines and answer engines, so use it selectively.
- Server-level blocks. For Apache,
.htaccessrules can deny access by user-agent. Our free .htaccess generator can build bot-blocking rules. For Nginx, equivalent rules go in the server configuration. Server-level blocks are enforced before your application even runs, which makes them efficient. - Hotlink and embed protection. If your concern includes images and media, hotlink protection stops other sites from embedding your files, and it reduces one channel through which your media ends up in other people's datasets.
What does not work: disabling right-click, overlaying transparent images, or other "copy protection" tricks aimed at browsers. They annoy your readers, they do nothing against a crawler that fetches your HTML directly, and they signal that you misunderstand the threat. Skip them.
Layer 6: Provenance — watermarks and content credentials
Provenance does not prevent training use, but it helps you prove what is yours after the fact. The Content Authenticity Initiative's C2PA standard — Content Credentials — lets you attach tamper-evident metadata to images and other media, recording who created them and how they were edited. If your work shows up in a dataset or a model output, credentials make the chain of ownership harder to deny.
Visible watermarks on images serve a similar purpose at a cruder level: they survive screenshots, re-uploads, and some forms of reuse. They also degrade the reader's experience, so use them on portfolio and stock-style images where theft is the main risk, not on every image on your blog. Invisible watermarking tools exist too, embedding signals that survive resizing and compression. Treat all of these as evidence tools, not prevention tools. Their job is to make your claim provable, not to make copying impossible.
What actually works: a realistic stack
No honest guide can promise that any opt-out is airtight. Here is what each layer realistically delivers:
- Robots.txt: respected by the major AI companies' documented crawlers. Ignored by bad actors. Necessary baseline.
- TDM reservation: the correct formal, machine-readable rights reservation, with real legal significance in the EU. Uneven adoption so far.
- ai.txt: a clear, parseable usage policy. Limited adoption, no legal standing. Cheap to add.
- Licenses and terms: establish your position for humans and courts. Do not stop automated collection on their own.
- Technical barriers: the only layer that actually stops a request. Require care to avoid blocking real visitors and search engines.
- Provenance: helps prove ownership after misuse. Does not prevent misuse.
The pattern is clear: declarations handle the compliant majority, enforcement handles the rest, and provenance covers what slips through. A site that publishes all six layers is in a far stronger position — technically, legally, and evidentially — than a site that publishes one.
A practical 30-minute opt-out plan
If you do all of this at once, it feels overwhelming. Do it in order, in one focused session. First ten minutes: write your one-paragraph AI-use policy — what is allowed, what is not, how to ask for a license — and publish it as a page on your site. Link it from your footer. Next ten minutes: update robots.txt to block the training crawlers you have chosen, and publish an ai.txt file stating the same policy. Final ten minutes: publish your tdmrep.json file at /.well-known/tdmrep.json, and add the TDM meta tag to your site's header. Then verify: open each file in a browser, test robots.txt, and check your server logs a week later to see which crawlers stopped visiting. Add firewall rules only for the crawlers that ignored the polite signals.
Revisit the stack twice a year. New crawlers appear constantly — a bot that did not exist in January can be a major scraper by June. Your server logs are the early-warning system: any user-agent fetching pages at high volume that you do not recognize deserves a lookup and, if it is an AI crawler you did not approve, a new rule.
FAQ
Does robots.txt legally stop AI companies from training on my content?
No, not on its own. Robots.txt is a voluntary technical signal with no settled legal status as a rights reservation in most jurisdictions. For a formal, machine-readable rights reservation, use the TDM Reservation Protocol, and back it with clear terms of use.
What is the TDM Reservation Protocol?
It is a W3C community protocol that implements the opt-out in Article 4 of the EU Copyright in the Digital Single Market Directive. You publish a tdm-reservation: 1 signal — via a tdmrep.json file, HTTP headers, or meta tags — to state that text and data mining of your content is reserved.
Is ai.txt an official standard?
No. ai.txt is a voluntary, plain-language policy file for declaring how AI systems may use your content. It has limited adoption and no legal standing, but it is cheap to publish and adds one more unambiguous declaration to your stack.
Can I opt out of AI training but still appear in ChatGPT and Perplexity answers?
Yes, if you are selective. Training crawlers (GPTBot, ClaudeBot, CCBot) and answer crawlers (OAI-SearchBot, Claude-Web, PerplexityBot) are different bots. Blocking the trainers while allowing the answer crawlers keeps your content out of training data while preserving visibility in AI answers — as long as those answer crawlers keep respecting the distinction.
What stops a crawler that ignores all of these signals?
Only enforcement: firewall rules, rate limiting, server-level blocks, or gating the content behind login. Declarations work on compliant crawlers. For the rest, you need technical barriers — and even those cannot stop content that was already collected or republished elsewhere.
Do watermarks prevent AI training on my images?
No. Watermarks do not prevent an image from being collected or trained on. Their value is provenance: they make your ownership visible and harder to deny after the fact. Use them as evidence tools, not as prevention.
