You can generate a robots.txt file for WordPress in about ten minutes, and you do not need to write it from scratch. The fastest route for most site owners is to build the file with a free online generator, check the result with a tester, and then put it live through your SEO plugin or by uploading it to your site root. WordPress also creates a basic version of the file on its own, so if you have never touched robots.txt, your site probably already has one. It just may not say what you want it to say.
This guide walks through every practical way to generate a robots.txt file on WordPress in 2026: the free generator route, the built-in virtual file, the Rank Math and Yoast editors, and the manual upload through your hosting file manager or FTP. You will also get a ready-to-use template that blocks AI training bots while keeping Googlebot and the search crawlers that bring you traffic. By the end, you will know how to test the file before it goes live and how to fix it when something goes wrong.
What a robots.txt file actually does
A robots.txt file is a small plain-text file that lives at yoursite.com/robots.txt. It tells search engine crawlers and AI bots which parts of your site they are allowed to visit. Each rule names a crawler, called a user-agent, and then lists paths to allow or block. A sitemap line at the bottom points crawlers to your XML sitemap so they can find your pages faster.
Three facts about robots.txt surprise most WordPress owners. First, it is a request, not a lock. Well-behaved crawlers such as Googlebot follow it, but a badly behaved bot can ignore it completely, so robots.txt is a traffic-management tool, not security. Second, it does not remove pages from search results on its own. Google says so plainly in its own documentation: to keep a page out of Google, use a noindex tag or password-protect the page. Third, the file is public. Anyone can open yoursite.com/robots.txt and read it, so never list private folders or file names there as if the file hides them.
On a WordPress site, a good robots.txt file does three useful jobs. It keeps crawlers out of the admin area and internal search pages, which saves crawl budget for the pages that matter. It points every crawler to your sitemap. And in 2026, it is the standard place to tell AI training crawlers whether they may use your content. That last job is why so many WordPress owners are generating a fresh robots.txt file this year.
Method 1: Generate it with a free online generator
If you want a correct file without learning the syntax first, start with a generator. This is the best option when your site has no robots.txt file yet, or when the one you have is years old and nobody remembers who wrote it.
Open the free robots.txt generator and work through the form. You will enter your domain and your sitemap URL, then pick the rules you want. A sensible starting set for WordPress looks like this:
- Allow everything by default. Your posts, pages, and images should stay open to crawlers.
- Block /wp-admin/. There is nothing for a search engine in your dashboard, and blocking it keeps bots from wasting requests there.
- Allow /wp-admin/admin-ajax.php. Some themes and plugins need this file to work for visitors, so it should stay open even though the rest of the admin folder is closed.
- Block /wp-includes/. These are WordPress core files, not content.
- Block internal search results. Paths such as /?s= create endless near-duplicate pages that add nothing to search results.
- Add your sitemap line. Point it at the sitemap your SEO plugin creates, usually sitemap_index.xml for Yoast and Rank Math, or wp-sitemap.xml for WordPress core.
Once the generator shows you the finished text, read it once before you copy it. Check that your domain is right in the sitemap line and that no folder you care about is blocked by mistake. Then copy the text or download the file. Before you put it live, paste it into the free robots.txt tester and check a few important URLs against it: your homepage, a blog post, and one URL you meant to block, such as /wp-admin/. The tester shows you in seconds whether each URL is allowed or blocked. Fix any surprise there, while the file is still on your screen, not after it is live.
The generator route has one habit worth keeping: save the final text somewhere safe, such as a note in your password manager or a file on your computer. Six months from now, when a plugin update or a new AI crawler makes you revisit the file, you will want to know what you started from.
Method 2: Understand WordPress's built-in virtual robots.txt
Here is something many WordPress owners never realise: WordPress does not ship with a physical robots.txt file. When a crawler asks for yoursite.com/robots.txt and no real file exists on the server, WordPress builds one on the fly and serves it. This is the virtual robots.txt. It exists only in memory, for that one request, and it disappears again afterwards.
The default virtual file is short and sensible. It allows all crawlers, blocks /wp-admin/ while allowing admin-ajax.php, and adds a sitemap line if WordPress can find one. For a small blog that has never thought about crawler control, that default is fine. The problem is that it says nothing about AI training crawlers, and it gives you no way to block a specific bot unless you change it.
One setting can quietly turn the virtual file into a disaster. Under Settings, then Reading, WordPress has a checkbox labelled "Discourage search engines from indexing this site". When it is on, WordPress serves a virtual robots.txt that blocks every crawler from every page. That setting exists for staging sites and sites still being built. On a live site it is almost always a mistake, and it is one of the most common reasons a WordPress site vanishes from Google. Open that page now and make sure the box is unchecked on your live site.
To see your current virtual file, open yoursite.com/robots.txt in a browser while logged out, or in a private window. What you see there is what crawlers see. If you like it, you may not need to change anything. If you want AI bot rules or tighter controls, move on to one of the editing methods below. Just remember the golden rule of the virtual file: it only appears when no physical robots.txt file exists on your server. The moment a real file is uploaded, the virtual one stops being served.
Method 3: Edit robots.txt with Rank Math
Rank Math is the easiest way to manage robots.txt if it is already your SEO plugin, because you never leave the WordPress dashboard. Rank Math edits the virtual file, which means it stores your text in the database and serves it whenever a crawler asks for robots.txt. It does not create a physical file on your server. That detail matters, and it is the source of the most common Rank Math confusion, which we will get to in a moment.
First, check that Rank Math is in Advanced Mode. The robots.txt editor only appears in Advanced Mode, so if you cannot find it, look for the mode toggle at the top of the Rank Math dashboard and switch it on. Then go to Rank Math SEO, then General Settings, then Edit robots.txt. You will see the current rules in an editor. Rank Math offers a visual rule builder where you add rows for the user-agent, the directive, and the path, and an Edit Raw tab if you would rather paste or type the full text yourself. For the template later in this guide, the raw editor is the faster choice: paste the whole block, check it once, and save.
Now the catch. If a physical robots.txt file already exists in your site root, Rank Math cannot override it. The web server always serves a real file first, so Rank Math greys out the editor and shows a notice instead of letting you save changes that would never appear. The fix is simple: open your hosting file manager, find robots.txt in the public_html folder, and rename it to something like robots-old.txt or delete it. Reload the Rank Math settings page and the editor will be available again. This is worth checking before you blame the plugin, because a leftover physical file from an old host migration is a frequent cause of "my edits do nothing".
After saving, open yoursite.com/robots.txt in a new browser tab and compare what you see with what you saved. Rank Math adds a sitemap line automatically, so do not be surprised to see one even if you did not type it. If the live file matches your saved text, you are done. If it shows something else entirely, a physical file is still in place somewhere, or a caching plugin or CDN is serving an old copy, and you should clear the cache and check again.
Method 4: Edit robots.txt with Yoast SEO
Yoast SEO takes a different approach from Rank Math, and it is worth knowing the difference before you choose. Where Rank Math edits the virtual file, Yoast writes a real, physical robots.txt file to your site root. That makes Yoast the better choice if you want a file you can see in your file manager, and it also means Yoast's editor keeps working even when a physical file already exists, because Yoast simply edits that file.
To use it, go to Yoast SEO in your dashboard, then Tools, then File editor. If the File editor menu is missing, file editing has been switched off on your site, usually by your host or by a setting in your WordPress configuration, and you will need to use the manual upload method instead. On the file editor screen you will see editors for robots.txt and for your server configuration file. If no robots.txt file exists yet, click the Create robots.txt file button. Yoast will create a starting file that allows all crawlers and adds a sitemap line pointing to your sitemap_index.xml. Edit the rules to match what you need, then click Save changes to robots.txt.
One warning applies to both plugins. The file editor screens sit next to powerful settings, and a wrong rule in robots.txt can block your whole site from search engines. Always save a copy of the current file before you change it, make one change at a time, and check the live file afterwards. If you use Yoast's editor and later switch to Rank Math, remember that the physical file Yoast created will still be there, quietly overriding Rank Math's virtual editor until you remove it. Pick one method and stick with it, so there is only ever one source of truth for your robots.txt file.
Method 5: Upload a physical robots.txt file by hand
The manual method is the most direct one, and it is the fallback when plugins cannot help, for example when file editing is disabled or when you manage the site for a client and want the file under version control. It also gives you complete control, because a physical file always wins over the virtual file and over any plugin editor.
Start by creating the file on your computer. Open a plain-text editor such as Notepad on Windows or TextEdit on a Mac set to plain-text mode, and paste your rules. Save the file with the exact name robots.txt, all lower case. Do not use a word processor such as Word or Google Docs, because they add hidden formatting that breaks the file. The file name matters: robots.txt with a capital letter, or a file saved as robots.txt.txt because Windows hid the extension, will not be found by crawlers.
Next, upload it to the root folder of your site. In your hosting file manager, that is usually the public_html folder, the same folder that holds wp-config.php. Open the file manager, go to public_html, click Upload, and select your robots.txt file. If you prefer FTP, connect with a client such as FileZilla, open the same root folder on the server side, and drag the file across. Either way, the file must sit in the root, not inside wp-content or any subfolder, or crawlers will never find it.
Once the file is in place, it takes over immediately. The virtual file stops being served, and Rank Math's editor will grey out if you use Rank Math. That is expected, not an error. From now on, the physical file is the only file that matters, so any future change has to be made to it directly. If you later decide you would rather manage robots.txt through a plugin, delete or rename the physical file first, and the plugin will take over again.
If your site sits behind a CDN such as Cloudflare, clear the cache for the robots.txt URL after uploading, or the CDN may keep serving the old copy to crawlers for hours. The same applies to caching plugins: clear the cache, then open yoursite.com/robots.txt in a private browser window and confirm the new rules are there.
A ready-to-use robots.txt template for WordPress
Here is a complete template you can use as a starting point. It opens your content to search crawlers, closes the admin and system folders, blocks internal search pages, and tells four well-known AI training crawlers to stay out. Replace example.com with your own domain in the sitemap line, and use your real sitemap URL. If your SEO plugin creates sitemap_index.xml, use that. If you rely on WordPress core, use wp-sitemap.xml.
User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /?s= Disallow: /search/ Disallow: /wp-login.php Disallow: /readme.html Disallow: /license.txt Sitemap: https://example.com/sitemap_index.xml User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: /
Notice what the template does not block. There is no rule against Googlebot, so Google keeps crawling and indexing your site exactly as before. There is no rule against OAI-SearchBot, the crawler OpenAI uses for ChatGPT search. Blocking GPTBot stops OpenAI from using your content to train its models, but OAI-SearchBot is a separate crawler, and leaving it allowed keeps the door open for your pages to be cited in ChatGPT answers. The two bots have different jobs, and your robots.txt file can treat them differently. If you want the full picture of which AI crawlers to allow and which to block, read our guide to blocking AI crawlers on WordPress before you add more bot rules.
A few lines in the template deserve a quick explanation. The rule for /?s= blocks WordPress internal search result pages, which are near-duplicates that waste crawl budget. The /wp-login.php and readme.html lines keep crawlers out of pages that have no value in search results. And the sitemap line at the end of the first group points every crawler, including the AI ones you allow, to your sitemap. The four bot groups at the bottom each contain a single Disallow: / rule, which blocks that bot from the entire site. That is the strongest statement robots.txt can make, and it is the right one for training crawlers when you have decided you do not want your content used to train AI models.
Do not block your uploads folder. Some older templates include a rule against /wp-content/uploads/, which stops search engines from finding your images. Your images can bring traffic through image search, and Google needs to see them to understand your pages, so leave that folder open. For the same reason, never block your CSS and JavaScript files. Google renders pages to judge their quality, and it cannot do that if your stylesheets and scripts are blocked.
If you want the official rules behind all of this, Google's robots.txt specification explains exactly which directives crawlers must follow, and Rank Math's guide to editing robots.txt walks through the plugin editor step by step.
Test the file before you rely on it
Do not assume the file is live just because you saved it. Open yoursite.com/robots.txt in a browser and check that it shows your new rules, then run it through a robots.txt tester to catch syntax errors. Google Search Console also has a robots.txt report under Settings that shows the file Google fetched and flags errors it found — check it a day after publishing, since Google caches robots.txt for up to 24 hours.
Frequently asked questions
Does WordPress create a robots.txt file automatically?
Yes. If you have no physical robots.txt file in your site root, WordPress serves a virtual one that allows all crawlers and points to your sitemap. It is a sensible default, but it says nothing about AI training bots, so you still need to edit it if you want GPTBot or ClaudeBot kept out.
Will blocking GPTBot hurt my Google rankings?
No. GPTBot is OpenAI's training crawler and has nothing to do with Google Search. Googlebot is a separate crawler with its own user agent, so a rule that disallows GPTBot leaves your Google rankings untouched. The bot to keep allowed for ChatGPT citations is OAI-SearchBot, not GPTBot.
Should I use Rank Math or Yoast to edit robots.txt?
Use whichever SEO plugin you already run, but know they work differently: Rank Math edits WordPress's virtual robots.txt, while Yoast's file editor creates and edits a physical robots.txt file in your site root. A physical file overrides the virtual one, so if you switch plugins later, check which file is actually live. Do not install a second SEO plugin just for this — running two SEO plugins side by side causes more problems than it solves.
How long does it take for crawlers to respect a new robots.txt?
Google caches robots.txt for up to 24 hours, so changes usually take effect within a day. AI crawlers are less predictable: some pick up the new file on their next visit, others take days. Check your server logs a week after publishing to confirm the bots you blocked have stopped appearing.
Do I need a physical robots.txt file, or is the virtual one enough?
For most sites the virtual file is enough. Upload a physical file only if you need something the plugin editors cannot do — for example, different rules per environment, or rules that must survive a plugin being deactivated. Remember that a physical file in the site root overrides the virtual one completely.
