Last week I wrote about llms.txt: nobody confirms it works, it costs nothing to have it, and whoever tells you it's indispensable is probably selling something. Today it's the opposite. Today I'm talking about something that actually works, that has been working for over 20 years, and that most websites have broken without knowing it. It's the sitemap. And yes, I know that sounds like the most boring topic in the world.
What a sitemap is (without overcomplicating it)
A sitemap is, literally, a map of your site. An XML file that lists all the important pages so that whoever reads it doesn't have to discover them one by one by following links. Think of it like a book's table of contents: you don't read the index to understand the book, but without it you'd have to flip through every page to find what you're looking for.
Google has been using sitemaps since 2005. It helps the crawler find your new pages faster, especially if your site is large or if your content isn't well linked internally. Imagine publishing a new blog post with no links pointing to it from anywhere else on the site. Without a sitemap, Google might take weeks to discover it. With a sitemap, it sees it in days.
That's the version everyone knows. The 2026 version is different, and that's why I'm writing this today.
What changed: AI crawlers are reading it too
[An analysis of 48 days of server logs](https://www.wislr.com/articles/ai-bot-behavior-log-analysis/) found something striking: ClaudeBot — Anthropic's crawler — and GPTBot — OpenAI's — both started requesting the sitemap for the first time on the same day. It's a single site, so it doesn't prove a rule. But it is a fairly clear hint about where the way those platforms discover content is heading.
In other words: the sitemap stopped being Google's exclusive territory. It's now one of the first things any system checks when it wants to understand your site's structure — whether it's a traditional search engine, a generative AI crawler, or an automated auditor.
This matters because ChatGPT, Claude, and Perplexity don't work like Google. They don't have a dynamic index updating every hour. When a language model decides which sources it knows and which ones it draws from, part of that process comes from what its crawlers were able to discover and process. If your sitemap leads them to broken pages, content that no longer exists, or a structure that was never updated since the site launched, that's the first impression they get of you. And it's a bad one.
The double-edged sword nobody wants to look at
Here comes the part nobody likes to hear: the sitemap can work against you.
If your site has poorly built pages, duplicate content, old campaign landing pages from promotions you no longer run, previous versions of pages that were never properly removed, or URLs that simply don't exist anymore — all of that can be in your sitemap. And if it's in your sitemap, any crawler that reads it will try to visit them.
I see companies that haven't checked their sitemap in years. The site was redesigned twice, the team changed, the agency that built it is gone, and the sitemap is still pointing to URLs from 2019 that return a 404 error. Nobody knows because nobody checks. Nobody checks because nobody looks at it. And nobody looks at it because it's tucked away at a URL ending in «/sitemap.xml» that 95% of site owners have never visited in their lives.
My opinion, not necessarily a gentle one:
The sitemap is exactly the kind of technical thing that the digital marketing industry ignores because it's not glamorous. There's no pretty dashboard. No colorful report. No trending blog post about it. But it's one of the most fundamental pieces of how search engines — and AI engines — understand your site's architecture.
I constantly see situations where serious resources go into producing content, running campaigns, and redesigning sites, while the sitemap is still pointing to pages that no longer exist. It's like renovating your entire storefront and leaving the broken sign at the entrance.
What strikes me most is that in 2025 and 2026, with all the noise around GEO — Generative Engine Optimization — and how to appear in AI model responses, many companies are investing in sophisticated strategies while ignoring that their crawlers arrive at the site, request the sitemap, and get a welcome list of broken URLs. If you want to understand how GEO differs from traditional SEO, the SEO/GEO glossary covers it clearly.
The sitemap is no longer just for Google
ClaudeBot and GPTBot are actively requesting it to discover your site's structure. If yours has broken URLs or pages that no longer exist, that's the first impression they get of you. And there's no scheduled second visit.
What might be wrong in your sitemap right now
You don't need to be technical to understand the most common problems. The ones that show up most often are:
- 404 URLs: pages that no longer exist but are still listed in the sitemap.
- Redirected URLs: the sitemap should point to the final URL, not to one that redirects somewhere else.
- Noindex pages included: if a page has a «do not index» directive, it shouldn't be in the sitemap — it's a contradiction that confuses crawlers.
- URLs from previous versions of the site: leftovers from past redesigns that nobody cleaned up.
- Sitemaps that were never updated: generated automatically at launch and never touched again, even as the site changed multiple times.
The frustrating part is that many CMS platforms generate the sitemap automatically, which creates a false sense of security. The fact that it exists doesn't mean it's correct. WordPress, for example, has generated a sitemap by default since version 5.5 — but it includes everything: author pages, category archives, tag archives, internal search result pages. Things you probably don't want Google or AI crawlers visiting.
How to know if yours is fine (or broken)
The fastest way is to look for it. On most sites it lives at «/sitemap.xml» or «/sitemap_index.xml». If you can't find it there, check your domain's «robots.txt» file — the sitemap is usually referenced there.
Once you have it, the next step is validating it. And that's where most people stop, because it means manually checking hundreds or thousands of URLs. That's exactly why I built a sitemap validator inside AEON42: it finds your sitemap, checks every URL it lists, and tells you which ones point to pages that no longer exist or have issues. No signup, no account, in under a minute.
It's not magic. It's just automating something that would otherwise take hours by hand. The output is a concrete list of what's broken and what needs fixing.
What to do when you find the problems
Identify the broken URLs
Any URL returning a 404 in your sitemap is an immediate problem. Crawlers try to visit it, fail, and that sends a negative signal about your site's overall quality. Remove these from the sitemap or fix where they point.
Fix the redirected URLs
If a URL in your sitemap ends up redirecting to another URL, update the sitemap to point directly to the final destination. A redirect isn't a critical error, but it's a signal that the sitemap isn't clean.
Remove pages you don't want indexed
Noindex pages, internal search result pages, admin pages, or archive pages that don't contribute real content — they should be out of the sitemap. The sitemap should be a curated list of your site's best content, not an inventory of everything that exists.
Set up automatic regeneration correctly
If you use a CMS, check when and how the sitemap updates. Ideally, it should regenerate when you publish new content or make structural changes to the site. A static sitemap that never changes isn't very useful on a site that does change.
Submit it to Google Search Console
Once the sitemap is clean, submit it manually from Google Search Console. This tells Google there's an updated sitemap and accelerates the re-crawl process. For AI crawlers there's no equivalent panel today, but a well-formed sitemap at the standard URL is enough for them to find it.
The question nobody asks: is it worth the time?
Always. Not because it's urgent in the sense that your site will collapse today if you don't do it. But because it's one of the few technical things with a low implementation cost and real potential impact.
Think about it this way: if an AI crawler visits your site today to decide whether you're a source worth citing, and the first thing it does is read your sitemap and find 40 URLs returning 404s, that crawler already has a negative signal about your site's quality. Maybe it's not decisive. Maybe other factors compensate for it. But it's a signal you don't need to give away.
If you already have a serious content strategy, if you're already working on technical SEO and search engine positioning, if you're already thinking about how to appear in AI model responses — then the sitemap is the kind of thing you can't afford to have neglected. It's the foundation, not the ceiling.
And if you're not doing any of that yet, the sitemap is a good starting point because it gives you an honest diagnosis of your site's current state. What you find there will tell you a lot about how well or how poorly the rest is maintained.
If you'd rather have someone review this with you and propose what to fix first, the Play.Interactive team can help — with or without a broken sitemap.
Frequently asked questions
Not necessarily, but most sites built in the last 10 years do — even if the owner has never seen it. CMS platforms like WordPress, Shopify, Wix, or Squarespace generate one automatically. The fact that it exists doesn't mean it's properly configured or up to date.
Yes. There's documented evidence in server logs that both GPTBot and ClaudeBot request the sitemap to discover a site's URLs, rather than only following internal links. It's the same behavior Googlebot has had for decades, and AI crawlers are now adopting it as well.
Finding the problems can take under a minute with an automated validator. Fixing them depends on how many broken URLs you have and how easy it is to modify the sitemap in your CMS. For a well-maintained site of 50-100 pages, an afternoon is enough. For larger sites with years of accumulation, it can be a multi-day project.
No. Removing a URL from your sitemap doesn't make it disappear from Google or lose its current rankings. The sitemap tells Google where to look — it doesn't determine which pages get indexed. You can clean your sitemap with confidence without affecting pages that are already ranking well.
The file is the same. There's no different format for AI crawlers versus Googlebot. The difference is in how clean and accurate that file is: a sitemap with broken URLs hurts all crawlers equally, and a well-maintained one benefits all of them equally. There's no special configuration to do for each one.