Skip to content

Armel Silvère Dongou

Finance, écriture et entrepreneuriat

How to Use a Sitemap to Crawl a Website Efficiently?

A sitemap lists a website’s URLs in a structured file, most often in XML format. Search engines read it to…

Femme analysant un sitemap imprimé sur un bureau en bois dans un bureau à domicile moderne

A sitemap lists a site’s URLs in a structured file, most often in XML format. Search engines read it to discover pages, but for users, this same file becomes an exploration tool: it helps map a site, identify its main sections, and measure the gap between what is published and what is actually accessible through navigation.

XML, HTML and TXT sitemaps: what each format reveals about a site

Not all sitemaps provide the same information. The format determines what can be extracted during an exploration.

Format Primary audience Usable data Limitations for exploration
XML Search engines (Google, Bing) URLs, last modified date (lastmod), image/video extensions Does not always include orphan pages or resources blocked by robots.txt
HTML Human visitors Visible site structure, clickable links to each section Often incomplete on large sites and updated manually
TXT Search engines Raw list of URLs, one per line No metadata (no dates or priorities)

XML is still the most useful format for a systematic exploration. It provides the lastmod date, which—when it reflects a genuine change—makes it possible to sort pages by freshness. Google says it uses this date when it consistently corresponds to an actual change in the content.

Google does not take the changefreq and priority tags into account, however. Treating them as reliable indicators would distort the analysis. A TXT sitemap, on the other hand, provides only a list of addresses with no context: it is a starting point, not an auditing tool.

Web developer standing in front of a whiteboard with a hand-drawn sitemap in a startup office

Compare the sitemap with pages that are actually accessible

The most productive use of a sitemap is not to read it passively. Instead, compare it with other data sources to identify discrepancies.

Cross-referencing the URLs in an XML sitemap with those discovered by an internal crawl (Screaming Frog, Xenu or any other tool) reveals two types of issues. First, pages included in the sitemap but missing from the internal link structure: these URLs exist, but no navigation links point to them. Second, pages linked to from the site but missing from the sitemap, which may indicate that they were not declared or that the file was generated incorrectly.

To illustrate this method, the Europe Entreprises sitemap shows how an online directory organizes its URLs by category and subdomain, making it easier to understand the site structure before even browsing the site.

In Google Search Console, the Sitemaps report shows whether the file has been read and how many URLs it contains. The indexing report can then be used to review submitted URLs that are not indexed, along with the reasons provided. This comparison turns the sitemap into a diagnostic tool, rather than a simple declaration.

The most common discrepancies and what they mean

  • URL in the sitemap but not indexed, with the status “Discovered – currently not indexed”: Google knows about the page but has not yet deemed it worth crawling. A lack of internal links to the page is often the cause.
  • Indexed URL missing from the sitemap: the page was discovered through links, but the XML file does not reference it. The sitemap is probably outdated or generated using overly restrictive filters.
  • URL in the sitemap returning a 404 error or a 301 redirect: the file was not updated after content was deleted or moved. Every URL that returns an error wastes the crawl budget allocated to the site.

Specialized exploration: images, videos and multilingual versions in the sitemap

The XML sitemap protocol supports extensions that expand the file’s coverage beyond standard pages. Image and video extensions declare resources that search engines might not discover by crawling the HTML code alone.

An image sitemap associates each visual with the URL of the page hosting it. For an e-commerce site or portfolio, checking this extension reveals whether product images are properly declared or remain invisible to Google Images.

The hreflang extension lists the relationships between language versions of the same page. On a multilingual site, exploring the hreflang sitemap makes it easy to spot inconsistencies: a French page pointing to an English version that no longer exists, or a Spanish version missing from the declaration.

Hands on a mechanical keyboard with an XML sitemap file open in a code editor on an ultrawide screen

Check the XML file’s technical validity

Before using a sitemap for exploration, make sure the file is valid. A malformed sitemap (an unclosed tag, incorrect encoding, or a relative URL instead of an absolute one) will be ignored by search engines and skew any manual analysis.

  • Each URL must be absolute and begin with the protocol (https://). Relative URLs do not comply with the sitemaps.org standard.
  • The file must not exceed the limit of 50,000 URLs per individual sitemap. Beyond that, a sitemap index splits the list across multiple files.
  • Adding a declaration to the robots.txt file (a Sitemap: line followed by the file URL) ensures search engines can find it without going through Search Console.

Sitemaps and internal linking: complementary tools, not interchangeable

A common mistake is to assume that a complete sitemap makes a strong internal link structure unnecessary. A sitemap signals that a page exists; it does not make the page important. Google discovers a URL through the sitemap, but assesses its relevance based on the internal links pointing to it, the content it offers, and its depth within the site’s architecture.

A page listed in the sitemap but accessible only four clicks from the homepage, with no contextual links from other content, is still given little weight. A sitemap solves a discovery problem, not a structural one.

Exploring a site through its sitemap is therefore best combined systematically with an analysis of its internal linking. A clean, up-to-date and technically valid sitemap remains the most reliable starting point for understanding what a site wants to show search engines—and what it forgets to show them.

How to Use a Sitemap to Crawl a Website Efficiently?