AI Crawlers and US Copyright Law in 2026: What Site Owners Can Actually Do
AI Search

AI Crawlers and US Copyright Law in 2026: What Site Owners Can Actually Do

 Rahul Sharma  Nov 08, 2026  18 min  ~3,507 words

If an AI company trains its models on your website's content without asking, can you stop it? In the United States, the honest answer in late 2026 is: partly. Three federal judges ruled on AI training and fair use in 2025, and they did not agree with each other. One said training was not fair use. Two said it was. The biggest case of all, the New York Times against OpenAI and Microsoft, still has no ruling. This article explains what each case actually decided, in plain English, and what practical steps you can take today as a site owner.

One thing to get straight up front: none of this is legal advice. Copyright law in this area is moving fast, the rulings so far come from trial courts, and your situation depends on your content, your contracts, and your jurisdiction. Treat this as general information that helps you ask a lawyer the right questions.

Fair use, in plain English

Fair use is the part of US copyright law that lets people use copyrighted work without permission in some situations. It is not a single rule. It is a test with four factors, set out in Section 107 of the Copyright Act, and a judge weighs all four together.

The four factors are: the purpose and character of the use (is it transformative, does it add something new, or does it just copy?), the nature of the copyrighted work (facts get less protection than creative expression), the amount used (a short quote is different from copying a whole book), and the effect on the market for the original work (does the use replace the original, or steal its customers?).

That last factor, market effect, turned out to be the one that mattered most in the AI training cases. It is also the reason the 2025 rulings point in different directions. In one case the AI tool was built to compete directly with the content owner. In the others, the AI model was a general-purpose tool, and the copyright owners could not show that it was taking their customers. Same legal test, very different facts, very different results.

Another point that trips people up: fair use is a defence, not a permission slip. An AI company does not get to declare its own training fair use. Only a court can decide that, case by case. And until an appeals court rules on generative AI training, which has not happened yet, every trial-court decision is only binding on the parties in that case. Other judges can look at the same question and reach a different answer.

The four rulings that matter

Four cases define where the law stands today. Three were decided in 2025. The fourth is still waiting for a decision. Here is what each one actually held, without the hype.

1. Thomson Reuters v. ROSS: fair use rejected

On February 11, 2025, Judge Stephanos Bibas of the federal court in Delaware ruled that ROSS Intelligence's use of Thomson Reuters' Westlaw headnotes to train its AI legal research tool was not fair use. The headnotes are the short editorial summaries of legal points that appear at the top of court opinions in Westlaw. ROSS used thousands of them, obtained through a third party, to build a tool that competed directly with Westlaw.

The judge's reasoning came down to two things. First, the use was barely transformative: ROSS used the headnotes for the same purpose Thomson Reuters did, to help people find relevant law. Second, the market harm was direct and obvious: ROSS was building a cheaper rival to Westlaw, aimed at the same paying customers. Copying a work to build a competing substitute is the classic case against fair use, and calling the copying "AI training" did not change that.

This was the first US federal court decision to address fair use in the context of AI training data, and it was a win for content owners. But its limits matter as much as its result. ROSS's tool was not generative AI. It returned passages from existing judicial opinions instead of creating new text. The content was paywalled, and the tool was a direct competitor. The ruling says little about a general-purpose chatbot trained on public web pages.

The story did not end there. On September 29, 2026, the Third Circuit Court of Appeals affirmed the decision, making it the first appellate ruling on AI training and fair use. The appeals court stressed the narrow facts: a non-generative tool, built to compete with the source it copied. It is an important precedent, but it is not a blanket ruling that AI training is infringement. The American Bar Association's analysis calls it a narrower precedent than the headlines suggested, and that caution is worth remembering.

2. Bartz v. Anthropic: training is fair use, piracy is not

On June 23, 2025, Judge William Alsup in the Northern District of California ruled on the first case to test whether training a generative AI model on copyrighted books is fair use. Three authors, Andrea Bartz, Charles Graeber, and Kirk Wallace Johnson, had sued Anthropic, the company behind Claude.

Judge Alsup split Anthropic's conduct into three separate uses and gave three different answers. Training Claude on books Anthropic had lawfully acquired was fair use: he called the training purpose highly transformative, because the model learns patterns from the text rather than republishing it. Scanning print books Anthropic had bought, to turn them into digital files, was also fair use, as a format change of copies it owned. But downloading more than seven million pirated books from shadow libraries like LibGen to build a permanent central library was not excused at all, and that part of the case went toward trial on damages.

Faced with that trial, Anthropic settled the piracy claim for at least $1.5 billion, roughly $3,000 per work across about 482,000 books, in what the court called the largest copyright class action settlement in history. Final approval came on July 20, 2026. The settlement covered only the piracy claim. It did not undo the judge's ruling that training itself was fair use, and it is not a licence to train on copyrighted works.

For site owners, the lesson of Bartz is about how the content was obtained. The court drew a sharp line between lawfully acquired material and pirated material. A crawler that visits your public pages is not pirating anything in the sense this case used the word. But the ruling also shows that courts will look hard at the source of training data, and that the "we just downloaded it" defence has real limits.

3. Kadrey v. Meta: fair use, but only because the authors could not prove market harm

Two days after Bartz, on June 25, 2025, Judge Vince Chhabria in the same California court ruled in Kadrey v. Meta. Thirteen authors, including Sarah Silverman and Junot Diaz, had sued Meta over the use of their books, downloaded from shadow libraries, to train its Llama models. Judge Chhabria granted Meta summary judgment on fair use.

Like Judge Alsup, he found the training use highly transformative: the books were written to be read, while Llama uses them to learn patterns in language. But his decision turned on the fourth factor, market harm, and it turned on the evidence, not the principle. The authors, in the judge's view, had not shown that Llama substitutes for their books or dilutes the market for them. He was explicit that his ruling does not stand for the proposition that Meta's use of copyrighted material to train its models is lawful in general, and he warned that a better-argued case, with real evidence of market harm, might defeat a fair use defence.

The court also left other claims alive. Neither side had asked for summary judgment on the claim that Meta distributed the authors' works by re-uploading pirated files through BitTorrent while building its training data, so that issue stayed open. Kadrey is best read as a warning to both sides: transformative purpose helps AI companies, but the market-harm question is still wide open, and the next plaintiffs will come with better evidence.

4. New York Times v. OpenAI and Microsoft: still undecided

The case most likely to define the law for publishers is the one with no ruling yet. The New York Times sued OpenAI and Microsoft on December 27, 2023, alleging that millions of its articles were copied without permission to train ChatGPT and related systems. The case, now consolidated with other publisher and author suits before Judge Sidney Stein in New York, is the biggest test of AI training fair use for news content.

On April 4, 2025, Judge Stein rejected most of the companies' motions to dismiss, letting the core claims proceed. In September 2026, both sides filed cross-motions for summary judgment on fair use, and the US Department of Justice filed a statement of interest arguing that training AI models on copyrighted material can be fair use. As of November 2026, the court has not ruled. Nothing has been decided, and no finding of infringement or wrongdoing exists. A plain-English explainer of the 2025 rulings puts it well: read closely, Bartz and Kadrey point in different directions, and neither one is the final word.

The Times case matters because it tests arguments the book cases did not. It is about news articles, not books. It includes claims about model outputs that reproduce Times content nearly verbatim, not just about the training copies. And the plaintiff is a publisher whose business is selling the very content being copied. Whatever Judge Stein decides, the losing side will almost certainly appeal, so a final answer is still years away.

What the rulings add up to

Read together, the three decided cases point to a pattern, even if they do not add up to a rule yet.

First, training a general-purpose model looks more like fair use than training a direct competitor. The ROSS tool lost because it was built to replace Westlaw. Claude and Llama won, for now, because they are general tools that do not substitute for any one author's books. If your content is the product itself, and an AI tool is trained on it to offer the same product, your position is stronger than if your content is one small part of a giant training set.

Second, market harm is the battleground. Courts keep coming back to whether the AI use takes money or readers away from the copyright owner. That is good news and bad news for site owners. It is good news because a site whose traffic or subscriptions fall after AI tools start answering its questions has a real argument. It is bad news because the argument needs evidence: traffic data, revenue records, examples of AI answers replacing visits to your pages. "It feels unfair" is not a legal theory.

Third, how the data was obtained matters. Pirated copies lost in Bartz even though training on lawful copies won. Crawlers that respect the rules and copy public pages are in a different position from bulk downloads of paywalled or pirated material. Keep records of what was taken from your site and how, if you can.

Fourth, nothing is settled. All three 2025 rulings came from trial courts. The one appeals ruling, in Thomson Reuters, was deliberately narrow. The US Copyright Office published a report on generative AI training in May 2025 that also raised market dilution concerns. With more than a hundred AI copyright lawsuits working through US courts, the law will keep moving, and today's answer may not be next year's.

Why robots.txt is a request, not a legal shield

Many site owners assume that blocking a bot in robots.txt gives them a legal right to stop it. It does not, at least not under US law. Robots.txt is a technical convention: a file that asks well-behaved crawlers not to visit certain pages. It creates no contract, and US copyright law has no rule that says ignoring robots.txt is infringement by itself.

That does not make robots.txt useless. Far from it. Most major AI companies say they respect it, and blocking training bots in robots.txt is still the single most effective practical step a site owner can take. But understand what it does: it signals your preference to companies that choose to honour it. If a company ignores it, your remedy is not automatic. You would still need to show infringement and defeat a fair use defence in court.

There is also a subtle point about evidence. A clear robots.txt opt-out, combined with a terms-of-use page that forbids AI training, helps establish that the copying was not authorised and that you objected. That can matter in a dispute, even if the file itself is not a legal instrument. Think of it as a "no trespassing" sign: the sign does not create the property right, but it makes it much harder for someone to claim they did not know.

And keep your opt-outs separate. Blocking a training bot like GPTBot is not the same as blocking a search bot like OAI-SearchBot, which powers ChatGPT citations. If you block both, you lose the citations along with the training. A careful robots.txt file names each bot and treats them differently.

What you can actually do: six practical levers

Here is what is realistically available to a site owner today, in rough order of effort.

1. Block training bots in robots.txt. This is the first step and the cheapest. Add rules that disallow the training crawlers: GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, and others. Keep Googlebot and the search crawlers allowed if you want search traffic and AI citations. Test the file after you publish it, because a typo can block the wrong bot. A free robots.txt generator can build the rules for you if you do not want to write them by hand.

2. Declare your position in an ai.txt file. The ai.txt proposal lets you state, in a machine-readable file at the root of your site, whether your content may be used for AI training. It is a new standard and not yet widely honoured, so treat it as a complement to robots.txt, not a replacement. It costs almost nothing to publish — a free ai.txt generator can build the file for you — and it strengthens the record that you objected.

3. Put it in your terms of use. Add a clear clause to your terms of use or terms of service stating that automated collection of your content for AI training is not permitted without a licence. This matters because terms of use are a contract with your visitors, and some courts have treated systematic scraping in violation of terms as a separate legal problem from copyright. Keep the clause short and plain. Legal language that nobody can understand helps nobody.

4. Consider licensing. Some large publishers have signed licensing deals with AI companies, getting paid for training access instead of fighting over it. That option is mostly open to big publishers today, but collective licensing and smaller deals are growing. If an AI company approaches you, a licence is worth considering: it converts an unauthorised use into a revenue stream. If nobody approaches you, a clear "contact us to license" page at least makes the door visible.

5. Act on verbatim outputs. If an AI tool reproduces your content nearly word for word, that is a different claim from training itself. Courts have not yet ruled on whether model output can infringe, but the Times case is testing exactly that theory. In the meantime, you can send a DMCA takedown notice to the AI company or to the platform hosting the output, and you can document the output with screenshots, dates, and the prompts used. Evidence collected now is useful later, even if you never file a claim.

6. Keep records. Save your server logs showing which AI crawlers visited your site and when. Note traffic changes after major AI product launches. Record examples of AI answers that reproduce or summarise your content. If the law ever gives you a claim, or a licensing negotiation ever opens, this evidence is what turns a complaint into a case. Most site owners who wish they had records wish they had started keeping them a year earlier.

What not to do: do not assume a single technical trick solves the problem. No robots.txt rule, no ai.txt file, and no terms clause can physically stop a determined scraper. The honest goal is to raise the cost of ignoring you, create a clear record of objection, and preserve your options.

A quick note on Europe

The picture is different in the European Union, where the law gives copyright owners an explicit opt-out from text and data mining for AI training: if you reserve your rights, in a machine-readable way such as robots.txt, miners must respect it. That is a stronger legal position than anything US law currently offers, and it is one reason many sites publish their EU opt-out separately. We cover the EU rules in detail in our companion guide to the EU text-and-data-mining opt-out.

What to watch next

Three things will shape the law from here. First, the Times case: Judge Stein's ruling on the cross-motions for summary judgment will be the first fair use decision about news content and about verbatim outputs, and it will be appealed whatever it says. Second, the other pending cases: with more than a hundred AI copyright suits in US courts, the next rulings will test different kinds of content, from music to images to code, and different kinds of harm. Third, Congress and the Copyright Office: the May 2025 Copyright Office report on generative AI training suggested the law may need updating, and legislative proposals keep appearing, though none has passed yet.

For site owners, the practical takeaway is to act on what you can control now and keep watching. Set your robots.txt opt-outs. Publish your terms. Keep your records. And revisit your settings every few months, because both the law and the crawlers change faster than most people expect.

Frequently asked questions

Is AI training on my website's content legal in the US?

It depends, and no court has given a final answer for general-purpose models. Two trial courts ruled in 2025 that training on lawfully acquired books was fair use, one court ruled that training a competing tool on paywalled headnotes was not, and the biggest case, NYT v. OpenAI, is still undecided. Until an appeals court rules on generative AI training, the honest answer is that it is contested.

Does blocking GPTBot in robots.txt stop OpenAI from training on my content?

It stops OpenAI's crawler from fetching your pages going forward, if OpenAI honours the rule, which it says it does. It does not remove content that was already collected, and it is a technical request rather than a legal right under US law. It is still the most effective first step available, and it creates a clear record that you objected.

Can I sue an AI company for training on my site?

You can, in principle, if you own the copyright in the content and can show infringement. But individual lawsuits are expensive, fair use is a strong defence for general-purpose training, and your case will be judged on its own facts, including whether the AI use harms your market. Most site owners are better served by technical opt-outs, terms of use, licensing, and documentation than by litigation. Talk to a copyright lawyer before filing anything.

What is the difference between the ROSS case and the Bartz case?

ROSS built a tool to compete directly with Westlaw using Westlaw's own paywalled summaries, and the tool was not generative. Anthropic trained a general-purpose model on lawfully acquired books. The courts treated those as very different uses: copying to build a substitute lost on fair use, while training a general model won, at least at trial level. The facts matter more than the label "AI training."

Does the EU text-and-data-mining opt-out protect my US site?

Not directly. The EU opt-out is a right under EU law, and it applies to uses covered by that law. But publishing a clear opt-out in robots.txt helps everywhere: it signals your position to crawlers worldwide, and it supports any claim you might make under US law that the copying was unauthorised. Many sites publish one robots.txt file that serves both purposes.

Should I block all AI crawlers, or only the training bots?

Only the training bots, in most cases. Blocking search crawlers like OAI-SearchBot, Claude-SearchBot, or PerplexityBot also removes your pages from AI search answers and citations, which is free visibility. Split your robots.txt rules: disallow the training bots, allow the search bots, and keep Googlebot allowed for normal search.

Disclaimer: This article is general information about US copyright law and AI training. It is not legal advice, and it does not create a lawyer-client relationship. Copyright law is changing quickly, and the cases discussed here may be appealed or overtaken by new rulings. If you are considering legal action or a licensing decision, consult a qualified copyright attorney in your jurisdiction.

Related Topics & Keywords

Share this article

Related Articles