Your robots.txt file just gained legal weight. In July 2026 the European Data Protection Board adopted Guidelines 03/2026 on web scraping in the context of generative AI, and the guidance tells AI developers to respect the opt-out signals on your site, including robots.txt, ai.txt files, and CAPTCHA. Publicly accessible no longer means free to process. If your website contains personal data, and most websites do, the GDPR now gives your crawler rules a regulator's backing. This guide explains what the rules actually say, what they do not do, and the practical steps to take on your own site.
First, the scope. The GDPR is a privacy law, not a copyright law. It covers personal data: names, photos, email addresses, comments, reviews, forum posts, and anything else that identifies a living person. It does not cover your product descriptions, your prices, or your generic blog content. If an AI crawler scrapes your site and your site has no personal data on it, the GDPR is not the tool for you, and you should use copyright opt-outs instead. Our guide to AI crawlers and EU copyright opt-out covers that side. But if your site has author bios, user comments, testimonials with real names, or customer reviews, the GDPR applies to that content, and this guide is for you.
What the EDPB's 2026 web-scraping guidelines actually say
The European Data Protection Board adopted Guidelines 03/2026 on 7 July 2026. The guidelines explain how the GDPR applies when personal data is scraped from the web to train generative AI. Three points matter most for site owners.
Public does not mean unprotected. The guidelines are explicit that personal information being publicly accessible does not remove GDPR requirements. A crawler that collects names and photos from public pages is still processing personal data, and it still needs a lawful basis under Article 6 of the GDPR.
Legitimate interest is the usual claim, and it has a test. AI developers most often rely on legitimate interest under Article 6(1)(f). The EDPB's earlier Opinion 28/2024 on AI models set out a three-step test for that claim: the interest must be legitimate, the processing must be necessary to achieve it, and the individual's rights must not override it. Scraping everything because it might be useful fails the necessity step. Indiscriminate collection is the weak point in most AI training pipelines.
Respect the signals on the site. This is the part that changes life for site owners. The EDPB's summary of the guidelines tells AI developers to stay away from websites that block scraping with robots.txt files, ai.txt files, or CAPTCHA. Your robots.txt is no longer just a polite request that a crawler may or may not honor. It is now a signal that a European regulator has told AI developers to respect. A developer that scrapes a site after the site said no in robots.txt is not just rude. It is building a record of ignoring a stated objection, and that record matters if a regulator ever asks questions.
Two honest caveats. The guidelines were open for public consultation until 30 October 2026, so details can still change, and guidelines are not law by themselves. They explain how the regulator reads the law, which is influential but not the same as a court ruling. Treat them as a strong signal of where enforcement is heading, not as a guarantee.
Your rights as a site owner, and as an individual
The GDPR gives rights to individuals, not to websites. That distinction matters, because the rights work differently depending on whose data is involved.
If your own personal data is on the scraped pages, you have the right to object under Article 21. When processing is based on legitimate interest, you can object on grounds relating to your particular situation, and the controller must stop unless it demonstrates compelling legitimate grounds that override your interests. That is a real right, but it is conditional, not absolute. The controller can argue back. The only absolute objection under Article 21 is against direct marketing, which AI training is not.
You also have the right to erasure under Article 17, the right to be forgotten. Here the practical problem is severe. Once personal data is baked into a trained model, deleting it is not like deleting a row from a database. There is no delete button for a model weight. Regulators know this, and it is one of the reasons the EDPB's guidance pushes developers toward data minimisation before training, rather than cleanup after it. Erasure requests against a trained model are hard to satisfy, which is exactly why prevention, through robots.txt and ai.txt, matters more than cure.
As a site owner, your leverage is different. You control the signals on your own site, and the EDPB has now told developers to respect them. You also control what personal data you publish in the first place. Minimising the personal data on public pages, for example by using first names only in testimonials or removing personal email addresses from public pages, is a GDPR measure in its own right.
The Digital Omnibus twist: the law may change
The legal picture is moving, and site owners should know about it. On 21 October 2026 the Council of the European Union agreed its negotiating position on the Digital Omnibus, a package that includes changes to the GDPR for AI development. The Council's text introduces a new Article 88bis covering processing in the context of developing and operating AI systems, and says that such processing may be carried out on the basis of legitimate interest under Article 6(1)(f).
Critics, including the EDPB and the European Data Protection Supervisor in a joint opinion, have questioned whether the new provision is needed at all, since the GDPR already allows legitimate interest where its requirements are met. Privacy advocates have gone further, warning that the changes could let companies reuse large stores of personal data for AI development on broader terms than the current law allows. The Council's text preserves the Article 21 right to object, so the objection route survives even if the provision becomes law.
The important fact for site owners is that none of this is law yet. The European Parliament has not agreed its position, and the three-way negotiation between the Council, the Parliament, and the Commission has not finished. The GDPR in force today is unchanged. Do not plan your site around a proposal that may never pass in its current form. But do watch it, because if Article 88bis survives, the legal footing for AI training could shift under guidance that is still fresh.
What to do on your own site: a practical checklist
Step 1: Block training crawlers in robots.txt. This is now your most important GDPR-adjacent action as a site owner, because the EDPB has told developers to respect it. Add a group for each major training crawler: GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, and cohere-ai. Keep the AI search crawlers, OAI-SearchBot and Claude-SearchBot, allowed if you want citations, since blocking them is a separate decision about visibility, not privacy. Our free robots.txt generator builds the file, and our AI crawler robots.txt template is a ready starting point.
Step 2: Add an ai.txt file. The ai.txt file declares your AI training policy in a machine-readable format, and the EDPB's guidance names it alongside robots.txt as a signal developers should respect. It states plainly whether your content may be used for training, and under what terms. Our free ai.txt generator builds one in a minute, and our ai.txt guide explains the format.
Step 3: Use noai and noimageai meta tags. These tags tell compliant AI systems not to use a page or its images for training. They work page by page, which makes them useful for the specific pages that carry personal data, such as team pages, testimonial pages, and community forums. See our noai meta tag guide for the details.
Step 4: Say it in your privacy policy. Add a short section stating that you do not consent to your site's content being scraped for AI training, and that you have set robots.txt and ai.txt accordingly. This is not a magic shield, but it creates a clear record of your position, and it tells your visitors where you stand.
Step 5: Minimise personal data on public pages. Remove personal email addresses from public pages, use first names only in testimonials where you can, and think twice before publishing photos with identifiable people alongside their full names. Less personal data in public means less personal data for a scraper to collect.
Step 6: Have a process for objections and erasure requests. If someone asks you to remove their personal data from your site, handle it promptly under your existing GDPR process. That is your obligation as a controller of your own site, and it is separate from what AI companies do with data they already scraped.
What the GDPR does not do
Honesty requires the limits. The GDPR does not remove your content from models that were already trained on it. It does not cover non-personal content, so your product copy and pricing are outside its scope. It does not automatically block a crawler. A robots.txt rule is still technically a request, and a scraper that ignores it faces no immediate technical barrier, only a growing legal and regulatory risk. And the GDPR is European law. It protects people in the EU and EEA, and it applies to companies that process their data, but a site owner outside Europe with no European visitors is in a different position.
None of that makes the guidance useless. It makes it a piece of a larger strategy. Robots.txt plus ai.txt plus meta tags plus a clear privacy policy plus data minimisation is a layered position. Each layer is weak on its own. Together they form a record that you objected, clearly and repeatedly, through every channel the regulators themselves named.
The X case: why regulators are paying attention
The reason this guidance exists is visible in the enforcement record. In August 2024, the privacy group noyb filed a complaint with nine European data protection authorities alleging that X had used the personal data of more than 60 million users in the EU to train its Grok AI system without consent. X had begun training on EU users' posts in May 2024, and the opt-out setting was buried seven steps deep in the interface and enabled by default. The Irish Data Protection Commission brought proceedings against X in the Irish High Court, and the case pushed the EDPB to call for a harmonised EU-wide framework for AI companies that train on social media posts.
The X case is about a platform training on its own users' data, which is not the same as a crawler scraping your website. But it shows the direction of travel. Regulators are treating AI training on personal data as a GDPR issue, they are acting on it, and they are telling developers to respect the signals site owners put in place. Your robots.txt is now part of that conversation.
FAQ
Does the GDPR let me stop AI crawlers from scraping my site?
Not directly. The GDPR protects personal data, not websites. But the EDPB's Guidelines 03/2026 tell AI developers to respect robots.txt, ai.txt, and CAPTCHA signals, so your crawler rules now carry regulatory backing for the personal data on your pages. For non-personal content, use copyright opt-outs instead.
What is the EDPB's position on web scraping for AI training?
The EDPB adopted Guidelines 03/2026 on 7 July 2026. They say publicly accessible personal data is still protected, legitimate interest must pass a three-part test, and AI developers should stay away from sites that block scraping with robots.txt, ai.txt, or CAPTCHA.
Can I object to my personal data being used for AI training?
Yes, under GDPR Article 21, if the processing is based on legitimate interest. The objection is conditional: the controller must stop unless it shows compelling legitimate grounds that override your interests. The only absolute objection is against direct marketing.
Does robots.txt now have legal force under the GDPR?
Not exactly. Robots.txt is still technically a voluntary request. But the EDPB has told AI developers to respect it, which means ignoring it is now a documented compliance risk rather than just bad manners. That is a meaningful change, even if it is not a law that says robots.txt is binding.
What is the Digital Omnibus Article 88bis?
It is a proposed GDPR provision in the EU Digital Omnibus package that would expressly allow processing personal data for AI development on the basis of legitimate interest. The Council agreed its position on 21 October 2026, but the Parliament has not, and it is not law yet.
Should I still use copyright opt-outs if the GDPR applies?
Yes. The GDPR covers personal data. Copyright covers your creative content. Use both: robots.txt and ai.txt for the crawler signals, and the EU text-and-data-mining opt-out for your copyrighted material. They protect different things.
This article is general information, not legal advice. If your site handles personal data at scale or you have received a regulatory inquiry, speak to a qualified privacy lawyer in your jurisdiction.
Start with the technical steps: build your file with the free robots.txt generator, declare your policy with the ai.txt generator, and read our EU copyright opt-out guide to cover the other half of the picture.
