Sitemap Index Files: How to Handle Sites with 10,000+ Pages

If your site has more than a few thousand pages, a single sitemap.xml will eventually hit a wall. The sitemaps.org protocol caps each sitemap file at 50,000 URLs and 50 megabytes uncompressed — and the real trouble starts long before that, when one giant file becomes slow to generate, slow for crawlers to download, and a single point of failure. A sitemap index file solves this by acting as a table of contents: one small XML file that points to many smaller sitemaps, each holding a slice of your site. Google and Bing both support them, and every large site — ecommerce stores, news publishers, marketplaces, job boards — should be using one.

The limits that force you to split

There are two hard limits in the sitemap protocol, and both apply per file:

• 50,000 URLs maximum per sitemap file.
• 50 MB uncompressed per sitemap file (the file itself may be gzipped for transfer).

But treating those numbers as your target is a mistake. In practice, problems show up much earlier. A 40,000-URL file can take minutes to generate dynamically, and if generation times out halfway, crawlers get a broken or stale file. One giant file also means every tiny content change forces a full regeneration. And when something goes wrong — a bad URL sneaks in, a template change breaks dates — you have to debug 50,000 lines at once.

Large sites feel this fastest. A store with 25,000 products plus category, brand, blog, and help pages can cross 40,000 URLs without anyone noticing. A publisher archiving a decade of articles can sail past 100,000. Splitting early, around 5,000–10,000 URLs per file, keeps everything fast, debuggable, and independently refreshable.

How a sitemap index file works

A sitemap index is just another XML file, but instead of listing pages, it lists sitemaps. Here's what it looks like:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemap-products.xml</loc>
    <lastmod>2026-10-07</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-posts.xml</loc>
    <lastmod>2026-10-06</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemap-categories.xml</loc>
    <lastmod>2026-09-28</lastmod>
  </sitemap>
</sitemapindex>

Each <loc> points to a regular sitemap file, and each optional <lastmod> tells crawlers when that file last changed. One index file can reference up to 50,000 sitemaps — enough for billions of URLs in theory. The index itself usually lives at /sitemap.xml or /sitemap_index.xml; the filename doesn't matter as long as you submit it and reference it correctly.

The full protocol rules are documented at sitemaps.org, and Google's implementation notes are in the Search Central sitemap docs. Both are worth bookmarking — the protocol rarely changes, so this is stable knowledge.

How to split your URLs the smart way

The single most important decision is how you slice. The lazy approach — sitemap-1.xml, sitemap-2.xml, chunks of 10,000 in database order — works technically but gives you nothing operationally. Split by content type instead:

• sitemap-products.xml — your money pages on an ecommerce site.
• sitemap-categories.xml — category and brand landing pages.
• sitemap-posts.xml — blog articles and guides.
• sitemap-pages.xml — static pages (about, contact, help).
• sitemap-news.xml — if you're a publisher, a separate Google News sitemap.
• sitemap-images.xml or sitemap-videos.xml — media-heavy sites.

Why content type beats alphabetical or numeric chunks: you can monitor indexing per section in Google Search Console's sitemap report. If your products index drops to 60% while posts stay at 95%, you know exactly where the problem is. You can also refresh sections on different schedules — products hourly, blog posts on publish, static pages monthly — instead of regenerating everything. And when a template bug corrupts one section, the rest of your sitemaps keep working.

For very large single-type collections (a marketplace with 400,000 listings), split further: sitemap-products-001.xml through sitemap-products-040.xml, grouped however your database makes easy — by ID range or by date. Just keep each file comfortably under the limits.

Automating sitemap index generation

Manual sitemaps die on large sites. Someone forgets to regenerate after a deploy, and three months later your sitemap lists products you discontinued in spring. Automation is non-negotiable, and you have three patterns to choose from:

1. Dynamic endpoints

Your application generates the XML on request, querying the database for published URLs. This is always fresh, but it puts load on your database every time a crawler fetches it — and crawlers fetch sitemaps a lot. Add aggressive caching (regenerate every hour, serve from cache or a CDN in between).

2. Scheduled static generation

A cron job rebuilds the files nightly or hourly and writes them to disk. Crawlers get instant static files, and your database never feels it. This is the most reliable pattern for large sites. Keep the last few generations so you can roll back if a build goes wrong.

3. CMS plugins

On WordPress, plugins like Rank Math and Yoast generate a sitemap index automatically (e.g., /sitemap_index.xml with per-post-type sitemaps). This is fine up to tens of thousands of URLs; beyond that, move to a custom solution.

Whichever pattern you pick, generate your sitemaps with a dedicated tool rather than hand-rolling XML when you can — our free XML sitemap generator handles the structure, escaping, and formatting so you don't ship malformed XML at 2 a.m.

Submitting your index to search engines

Generation is only half the job. Make sure crawlers actually find and use your index:

• Submit only the index file in Google Search Console (Sitemaps report), not the individual child sitemaps. Google follows the references automatically.
• Add a Sitemap directive to robots.txt: Sitemap: https://example.com/sitemap.xml. This is crawler-independent — Bing, DuckDuckGo, and others all read it. Use our robots.txt generator to add it cleanly, then verify the file with the robots.txt tester.
• Ping on major updates. You can notify Google with a simple GET request to https://www.google.com/ping?sitemap=YOUR_SITEMAP_URL after big content pushes (bulk product imports, migrations). It's a hint, not a command, but it costs nothing.
• Bing Webmaster Tools accepts sitemap index submission the same way GSC does.

What sitemap indexes can't do

A sitemap index is a discovery aid, not a ranking tool and not an indexing guarantee. Google is explicit: submitting a sitemap does not guarantee crawling or indexing. On large sites, this matters because of crawl budget — the number of pages Googlebot will crawl on your site in a given period.

A 500,000-page marketplace might only get 20,000 pages crawled per day. Your sitemap index tells Google those pages exist; it doesn't make Google care about all of them. So pair your index strategy with prioritization:

• Put your most important URLs in sitemaps with fresh lastmod dates.
• Keep thin, duplicate, or low-value pages out of sitemaps entirely — a sitemap is a list of URLs you want indexed, not a dump of your database.
• Fix internal linking so important pages are a few clicks from the homepage; crawlers follow links more than sitemaps.

7 mistakes that break sitemap indexes

1. Nesting index files. A sitemap index cannot reference another sitemap index — only regular sitemap files. Nested indexes are silently ignored.
2. Mixing hosts. All URLs in a sitemap must be on the same host as the sitemap file (www vs non-www, http vs https all count as different). Cross-domain sitemaps only work if you verify both domains in Search Console.
3. Lying with lastmod. Setting every URL's lastmod to today \"to get recrawled\" backfires — Google learns your dates are meaningless and stops trusting them. Only update lastmod when the page actually changed.
4. Including non-indexable URLs. Every URL in your sitemap should return HTTP 200, be indexable (no noindex tag), and be the canonical version. Redirects, 404s, and canonicalized duplicates waste crawl budget and erode trust in your sitemaps.
5. Extension mismatches. If you gzip your files, the <loc> in the index must point to the .xml.gz URL, and the server must serve it with the right content type.
6. Stale files. A sitemap whose lastmod says 2024 while the site changed daily signals neglect. Automate regeneration.
7. Forgetting the robots.txt line after migration. Teams rebuild sitemaps on a new domain and forget the Sitemap: directive — or worse, leave the old domain's URL in it.

Frequently asked questions

How many sitemaps can one index file reference?

Up to 50,000 sitemap files per index, per the sitemaps.org protocol. Combined with 50,000 URLs per file, a single index can theoretically cover billions of URLs. In practice, sites with over a few million URLs sometimes use multiple index files submitted separately.

Should small sites use a sitemap index?

No — it's unnecessary overhead. If you have fewer than a few thousand URLs, a single sitemap.xml is simpler and works perfectly. Reach for an index when one file gets hard to generate, slow to download, or crosses roughly 10,000 URLs.

Do I submit the index or the individual sitemaps to Google?

Submit only the index file in Search Console. Google automatically discovers and processes every sitemap it references, and the Sitemaps report will show you per-file stats for each child sitemap.

Can a sitemap index include sitemaps from another domain?

Only if you own and have verified both domains in Google Search Console. Otherwise, keep every URL on the same host as its sitemap file — the protocol requires it, and cross-host entries get rejected.

Does a sitemap index improve my rankings?

No. Sitemaps help with discovery and crawling efficiency; they are not a ranking factor. They help large sites by making sure new and updated pages get found quickly, which is a prerequisite for ranking — not a shortcut to it.

The bottom line

For large sites, the sitemap index isn't optional infrastructure — it's how you stay in control of what crawlers see. Split by content type, automate generation, submit only the index, keep lastmod honest, and keep junk URLs out. Do that, and your 100,000-page site becomes as easy for search engines to digest as a 100-page one.