AI is starting to change how people interact with websites, but a lot of the discussion around it still feels abstract. One practical area worth watching is how well websites can be understood and used by AI agents.

The Agentic Browsing audit in Chrome 146 gives SEOs, developers, and website owners a new way to test how ready their websites are for AI agents.

Chrome Lighthouse may now be moving in that direction with experimental checks related to agent-friendly browsing. Instead of only looking at traditional performance, accessibility, SEO, and best practices, these checks focus on whether a website is structured clearly enough for automated agents to read, navigate, and interact with.

That matters because AI assistants do not browse websites the same way people do. They rely heavily on clean HTML, accessible labels, predictable layouts, and machine-readable signals. If your site is difficult for assistive technologies or automated tools to understand, it may also be harder for AI agents to use reliably.

This does not mean every site needs to rush into a full “AI agent optimization” project. A lot of this is still early and experimental. But it does give site owners and developers a useful reason to revisit fundamentals: accessibility, layout stability, semantic HTML, and clear site structure.

In this guide, we’ll look at what the Agentic Browsing audit in Chrome 146 is intended to check, why it overlaps with technical SEO, and which fixes are actually worth prioritizing right now.

Table of Contents

Table of Contents

1. What is the Agentic Browsing audit in Chrome 146?
2. Why the Agentic Browsing audit matters for SEO
3. What the Agentic Browsing audit checks
4. How to use Chrome 146 to run the Agentic Browsing audit
5. What website owners should fix first
6. Agentic Browsing is an early signal, not an SEO shortcut

What is the Agentic Browsing audit in Chrome 146?

The Agentic Browsing audit in Chrome 146 is an experimental Lighthouse category focused on how well a website supports AI agent interaction.

Chrome Lighthouse is already widely used by developers and SEO professionals to test performance, accessibility, best practices, and SEO. With this newer audit category, Lighthouse adds another layer: agent readiness.

The Agentic Browsing audit appears to be a newer experimental category focused on a different question: can an AI agent understand and use your website?

That includes basic but important things like:

Audit Area	What It Looks At	Why It Matters
Accessibility labels	Buttons, links, menus, and form fields with clear names and roles	Helps both screen readers and AI agents understand what each element does
Layout stability	Whether page elements move while the page loads	Reduces frustration for users and makes automated interaction more reliable
Semantic HTML	Proper use of headings, buttons, forms, navigation, and page structure	Makes the content easier for search engines and AI systems to interpret
llms.txt	A machine-readable file that can summarize important site information	Gives AI systems a clearer starting point, although it is still optional
WebMCP	Early support for structured agent interaction with websites	Experimental now, but worth monitoring as AI browsing develops

An AI agent does not experience a website the way a human does. A person can visually scan a page, understand context from design, ignore small layout shifts, and guess what a button does based on surrounding content. An agent depends much more heavily on clean page structure, accessible labels, predictable layouts, and machine-readable information.

That means common technical issues can become bigger problems for agent-based browsing. For example, a button with no accessible name may still be understandable to a human because of visual context, but it may be unclear to an AI agent. A form field without a proper label may still look obvious on screen, but programmatically it can be difficult to interpret. A layout that shifts after loading can also make automated interaction less reliable.

According to Chrome’s documentation, the Agentic Browsing category is experimental and based on proposed standards. It does not work like the traditional Lighthouse Performance score, where everything is rolled into a weighted 0–100 number. Instead, it shows readiness checks and pass/fail signals related to machine interaction.

Before running the Agentic Browsing audit in Chrome 146, make sure your browser is updated and the experimental WebMCP flag is enabled.

For official technical details, you can review the Chrome 146 release notes.

Why the Agentic Browsing audit matters for SEO

At first, agentic browsing may sound like something only developers need to care about. But it has a clear connection to SEO.

Search engines have always rewarded websites that are easier to crawl, understand, and use. AI agents have similar needs. They need content that is easy to interpret, actions that are clearly defined, and pages that do not break when interacted with programmatically.

The overlap with technical SEO is strong. Accessibility, semantic HTML, layout stability, internal linking, and clear content structure all matter for both search engines and AI systems. A website that is messy for users is often messy for crawlers. A website that is messy for crawlers will likely be messy for agents too.

This is why the Agentic Browsing audit should not be treated as a separate “AI SEO trick.” It is more useful to see it as another layer of technical quality control.

If your website already has strong accessibility, stable layouts, well-labeled buttons, clean navigation, and crawlable content, you are probably closer to agent-friendly browsing than you think. If your site relies heavily on confusing JavaScript interactions, vague button labels, missing form descriptions, or unstable page elements, this audit may expose issues that were already hurting users.

There is also a visibility angle. As AI search tools and assistants become more common, websites that are easier for machines to understand may have an advantage in how their information is extracted, summarized, or acted upon. That does not mean the Agentic Browsing audit is a direct ranking factor. There is no reason to treat it that way right now. But it does highlight technical areas that already matter and are likely to matter more in the future.

This also connects with the broader shift toward AI search optimization. If you want to understand how SEO is changing around AI-generated answers, GEO, and AEO, read our guide on GEO, AEO and SEO for AI.

What the Agentic Browsing audit checks

The current Agentic Browsing audit focuses on a few practical areas. Some are already familiar to SEOs and developers. Others are newer and more experimental.

One area is accessibility. AI agents often rely on the accessibility tree to understand what is available on a page. If interactive elements do not have proper labels, roles, or names, the page becomes harder to navigate programmatically. This is another reason accessibility should not be treated as a side task. It directly affects how machines understand your website.

Another area is layout stability, especially Cumulative Layout Shift, also known as CLS. If a button or form field moves after the page loads, an agent may identify the element in one position and then fail to interact with it correctly after the layout changes. The same problem frustrates human users, especially on mobile devices.

The audit may also check for llms.txt, an emerging convention designed to give AI systems a concise, machine-readable summary of a website’s content. This file usually lives at the root of a domain, similar to how robots.txt does. However, it is important not to overstate its impact. Chrome’s Lighthouse documentation treats llms.txt as optional at the moment, not as a guaranteed AI visibility booster.

Then there is WebMCP, which is still early. WebMCP is related to giving AI agents a more structured way to understand and interact with website functionality. For most websites, this will not be something to fully implement immediately. It is worth monitoring, but accessibility, stable layouts, and clear HTML are more practical priorities today.

Sitemaps still matter in this bigger picture. A clean XML sitemap helps search engines discover and prioritize important pages, and it supports a healthier technical SEO foundation. For more details, read this guide on sitemap XML best practices.

How to use Chrome 146 to run the Agentic Browsing audit

Here is a practical tutorial you can follow to test your website.

First, check your current Chrome version. Open Chrome, click the three-dot menu in the top-right corner, go to Help, then click About Google Chrome. You can also type this directly into your browser:

chrome://settings/help

Chrome will show your current version and automatically check for updates. If Chrome 146 is available for your system, install the update and relaunch the browser.

If Chrome 146 is not available in your stable browser yet, you may need to use Chrome Beta or another Chrome release channel where the feature is available. Since this is an experimental area, availability can vary depending on your browser version and setup.

Once Chrome 146 is installed, open a new tab and go to:

chrome://flags/#enable-webmcp-testing

This should take you directly to the WebMCP testing flag. Change the setting from Default to Enabled. Chrome will ask you to relaunch the browser. Relaunch it before continuing, because the flag will not take effect until the browser restarts.

After Chrome reopens, go to the website page you want to test. It is a good idea to start with an important page, such as your homepage, a service page, a product page, or a lead-generation landing page.

Now open Chrome DevTools. On Windows or Linux, press F12 or Ctrl + Shift + I. On Mac, press Command + Option + I. You can also right-click anywhere on the page and select Inspect.

Inside DevTools, look for the Lighthouse tab. If you do not see it immediately, click the double-arrow icon to reveal hidden tabs. Once you are in Lighthouse, look at the list of audit categories. If the experimental setup is working, you should see Agentic Browsing as an available category.

Select the Agentic Browsing category. You can run it by itself, or you can keep the other Lighthouse categories selected if you want a broader technical report. Then click Analyze page load.

Lighthouse will reload the page and generate a report. When the report is ready, review the Agentic Browsing section carefully. Instead of looking only for a perfect score, pay attention to what the audit is actually telling you.

If the report flags accessibility issues, start there. Fix missing labels, unclear buttons, incorrect ARIA attributes, and form fields that do not have proper names. If it flags layout stability, check images, ads, embeds, fonts, banners, and any content that loads late and pushes the page around.

If the report mentions llms.txt, decide whether adding one makes sense for your website. A basic file can summarize your business, main content areas, and important URLs in a clean Markdown format. Do not treat it as magic, but it can be a useful machine-readable signal.

If the report flags WebMCP-related items, document the result and monitor the standard as it develops. For many businesses, this will be something to revisit later rather than an urgent fix today.

What website owners should fix first

The best way to approach this audit is to separate practical improvements from experimental ones.

Priority	What to Fix	Recommended Action
High	Missing accessibility labels	Add clear labels to buttons, links, icons, and form fields
High	Layout shifts	Define image dimensions, control font loading, and avoid late-loading banners
Medium	Weak page structure	Use semantic HTML for headings, navigation, buttons, and forms
Medium	Poor internal linking	Make sure important pages are easy to reach from other relevant pages
Low / Experimental	llms.txt	Add a simple file if you want to provide AI systems with extra context
Monitor	WebMCP	Keep an eye on the standard before investing heavy development time

Start with accessibility. Every important button should have a clear name. Every form field should have a proper label. Navigation should be understandable. Headings should follow a logical structure. Links should describe where they go instead of using vague text like “click here.”

Next, work on layout stability. Add width and height attributes to images where appropriate. Avoid injecting large banners above existing content after the page loads. Make sure fonts do not cause major shifts. Be careful with ads, popups, and third-party scripts that move important elements.

Then review your page structure. Use semantic HTML whenever possible. A real button should be a button, not a styled div. A navigation menu should be marked up clearly. A form should be structured in a way that both users and machines can understand.

After that, look at discoverability. Make sure your sitemap is clean, your internal links are logical, and your most important pages are not buried too deeply. If you decide to add llms.txt, keep it concise and useful. Do not stuff it with marketing language. Think of it as a short guide that helps a machine understand what your site is about.

Finally, keep an eye on WebMCP. It may become more important as browser-based AI agents mature, but it is still early. For now, most businesses will get more value from fixing accessibility, stability, and technical SEO basics.

Agentic Browsing is an early signal, not an SEO shortcut

The Agentic Browsing audit is not something to fear. It is also not something to blindly chase as the next big SEO hack. The Agentic Browsing audit in Chrome 146 should be treated as an early signal, not a replacement for traditional technical SEO.

The smarter approach is to treat it as an early warning system. It shows where the web is moving and reminds us that clean, accessible, stable websites are easier for everyone to use, people, search engines, assistive technologies, and AI agents.

For now, the most practical wins are clear: improve accessibility, reduce layout shifts, use semantic HTML, maintain a clean sitemap, and keep your technical SEO foundation strong. Those improvements help your website today, even if agentic browsing standards continue to change tomorrow.

And if you need help preparing your website for the next stage of search, Nona Digital Marketing helps companies leverage SEO and build stronger visibility online.

Over the past few weeks, if you follow SEO news or industry discussions on LinkedIn, one topic that has been showing up more and more is llms.txt.

Google has previously said that llms.txt does not serve much practical purpose. But more recently, Lighthouse, Google’s own tool for assessing website quality, started checking whether a site includes one.

Adding an llms.txt file probably will not earn your website more AI citations right now. Google Search has said it is not needed for AI visibility, while Lighthouse frames it more as an emerging convention that may help AI agents understand a site’s structure more efficiently.

So, let’s take a closer look at what llms.txt can actually do, where its limits are, and why the advice around it feels so mixed.

Table of Contents

1. How does the llms.txt file work?
2. Does the llms.txt actually work?
3. Why does the llms.txt file cause so much confusion?
4. So, should I create an llms.txt file for my website?

How does the llms.txt file work?

The llms.txt file is essentially a short, machine-friendly summary of a website’s content, usually written in Markdown.

llms.txt is still a proposed protocol. It has not been broadly adopted across the industry yet. That makes it very different from robots.txt, which is a long-established convention that most crawlers and systems already recognize and follow.

Does the llms.txt actually work?

There is currently no solid evidence that llms.txt improves rankings, AI visibility, or citation frequency. In fact, the evidence we have so far points in the opposite direction.

Google has not indicated that llms.txt is used by Google Search, AI Overviews, or AI Mode. Its official guidance for performing well in AI Search still focuses on the same fundamentals as traditional SEO: helpful content, accessible pages, good user experience, structured data where relevant, and proper preview controls, llms.txt is not listed as a requirement or recommendation in that guidance.

There is also no public confirmation from major AI companies such as OpenAI, Anthropic, Meta, or Google that they use llms.txt as a ranking, retrieval, or citation signal. A large SE Ranking study analyzing almost 300,000 domains found no clear relationship between having an llms.txt file and receiving more AI citations

So the safest conclusion is this: llms.txt has not been proven useful for SEO or AI visibility today. It may still become relevant in the future if major AI systems begin supporting it, but for now, it should not be treated as a ranking factor, a citation booster, or a replacement for strong, crawlable, well-structured content.

Why does the llms.txt file cause so much confusion?

The confusion around llms.txt comes from the fact that people often talk about it as if it has only one purpose.

On one side, there is the SEO debate. From that perspective, the evidence is clear: there is currently no proof that llms.txt improves rankings, increases AI citations, or makes a website more visible in Google Search, AI Overviews, ChatGPT, Claude, or any other major AI answer engine. For anyone expecting it to work like a shortcut to “AI SEO,” the file is likely disappointing.

On the other side, llms.txt is being discussed as a possible discoverability layer for AI agents. In this context, the goal is not to manipulate rankings or generate citations. The goal is simply to give machines a cleaner, easier-to-read summary of what a website contains, where the most important resources are, and how an AI system might navigate them.

That is why the signals seem contradictory. Google Search may say the file is unnecessary for SEO, while tools, plugins, documentation sites, and AI-focused developers may still experiment with it for other purposes. These are not necessarily opposing views. They are talking about different use cases.

So the real problem is not the file itself. The problem is the expectation around it.

If you treat llms.txt as an SEO ranking factor, there is no strong reason to believe it will help. But if you treat it as a lightweight, experimental way to organize your content for future AI agents, it starts to make more sense.

In short, llms.txt is confusing because it sits between two worlds: traditional SEO, where it currently has no proven impact, and the emerging world of AI agents, where its role is still being tested.

The fact that llms.txt is showing up in these places does not mean the protocol has been validated or widely adopted. It simply means that some tools and platforms are experimenting with it. And, as we know, not every suggestion from these platforms is automatically useful. Yoast’s writing recommendations are a good example of advice that often needs to be taken with a grain of salt.

Still, this creates noise. Site owners see one Google source suggesting that llms.txt is unnecessary, while another tool or document appears to recommend it. Naturally, that makes people wonder whether Google has changed its mind, whether the advice is inconsistent, or whether they are missing something important.

The simpler explanation is that these recommendations are about different use cases.

For SEO, llms.txt does not currently offer a proven benefit. But for certain types of websites, especially documentation-heavy sites, it may help AI agents understand and navigate content more easily.

That is why the guidance can look contradictory: one part of the conversation is about search visibility, while the other is about machine readability and AI-agent discoverability, agents on certain types of websites. That is why Google’s instructions change depending on the documentation you access.

So, should I create an llms.txt file for my website?

t really comes down to what you are trying to achieve.

1. If your goal is to improve SEO performance or appear more often in AI-powered search results, llms.txt is probably not worth prioritizing right now. There is no clear evidence that it helps with rankings, citations, or visibility.
2. If your goal is to make your site easier for AI agents to understand and navigate, then it may be worth testing. But even then, it is important to remember that llms.txt is still only a proposed protocol. We do not yet know how widely it will be adopted, which agents will read it, or whether it will become a meaningful standard.

So, in either case, expectations should stay realistic. Do not expect immediate results, and do not treat llms.txt as a high-priority technical task.

So far, its most practical use seems to be helping AI agents understand what is on a site and how to complete certain tasks. That means the value of llms.txt can vary a lot depending on the type of website. For a publisher, the return may be limited. For a software company, documentation platform, SaaS product, or ecommerce site with structured user journeys, the case may be stronger.

My recommendation is to look at your own site before deciding, ask whether llms.txt makes sense based on your current setup, the effort required to create and maintain it, and the potential upside if AI-agent adoption increases.

Also consider the risk of inconsistencies between your website and the Markdown version of your content. If the site changes but the llms.txt file does not, you may end up giving machines outdated or incomplete information.

And if you decide to implement it, treat it as an experiment. Define what you want to learn, how you will monitor results, and what would make the test worth continuing.

For teams that need SEO results now, there are usually more urgent priorities: crawlability, technical SEO, JavaScript rendering, internal linking, content quality, structured data, and overall site performance.

The same logic applies to AI-agent optimization. For most websites, fixing existing technical and content issues will be more valuable than investing too much time in a protocol that is still finding its place.

Want to keep up with what is actually working in SEO? Follow the Nona Digital Marketing blog, where I publish weekly analysis on the latest trends and changes in digital marketing.

Get in touch with us today and let’s turn this update into an opportunity for stronger search performance.

An XML sitemap is a file that lists all the important pages on your website and tells search engines like Google where to find them. Without one, search engines have to discover your pages on their own by following links, which works fine for small sites, but gets unreliable fast as your site grows.

For larger sites with hundreds or thousands of pages, a well-structured sitemap means Google spends its crawl budget on the pages that actually matter, not wasting time on duplicates or low-value content.

In this guide we’ll cover what an XML sitemap is, the different types, the right format to use, best practices, and exactly how to create and submit one. XML sitemaps are also a key part of technical SEO, especially for larger websites where crawlability and indexation matter.

Table of Contents

Table of Contents

1. What is an XML sitemap?
2. Which websites need a sitemap?
3. How to check if a website has an XML sitemap?
4. What are the types of sitemaps?
5. Why create an XML sitemap?
6. What are the benefits of sitemap.xml for SEO?
7. What information is included in a sitemap?
8. How to create a sitemap?
9. What to include in your sitemap.xml?
10. Which pages should be included in sitemap.xml?
11. Where should the Sitemap be placed?
12. How to submit a sitemap.xml to Google?
13. How do I update a sitemap.xml file?
14. What are the sitemap best practices for SEO?
15. What are the main problems related to sitemaps.xml?

What is an XML sitemap?

A sitemap.xml is a file that lists all the pages on your website in one place. Think of it as a map you hand directly to Google, instead of waiting for it to find your pages by following links, you’re telling it exactly where everything is.

There are two types: an XML sitemap for search engines, and an HTML sitemap for users. The XML version is the one that matters for SEO.

Which websites need a sitemap?

It is recommended that all websites have an XML sitemap to improve content indexing and discovery by search engines. Technically every website can benefit from a sitemap, but it becomes genuinely important in four situations.

• Large and complex websites: if you have hundreds or thousands of pages, Google has a crawl budget and won’t necessarily find everything on its own. A sitemap tells it which pages to prioritise.
• New websites: Google discovers pages by following links, and a brand new site has very few links pointing to it. A sitemap speeds up the initial indexing process considerably.
• Websites with frequently updated content: if you’re publishing new pages regularly, a sitemap signals those changes to Google faster than waiting for it to recrawl.
• Websites with hidden or hard-to-reach pages: if some pages are only accessible via JavaScript or have very few internal links pointing to them, they can get missed during a standard crawl. A sitemap acts as a safety net.

If your site is small, well-linked and rarely changes, a sitemap is still worth having, it just becomes less critical.

How to check if a website has an XML sitemap?

To check if a website has a sitemap, you can access the robots.txt file. The quickest way to check if a site has a sitemap is to go directly to the robots.txt file — just add /robots.txt to the end of any domain, like example.com/robots.txt. If there’s a sitemap, it’ll usually be referenced there with a line that starts with “Sitemap”.

You can also try going directly to these URLs, just swap out the domain for whichever site you’re checking:

• example.com/sitemap.xml
• example.com/sitemap.html
• example.com/sitemap_index.xml

Some sites use sitemap_index.xml instead of sitemap.xml

Google has a limit of 50,000 URLs per sitemap file. For a small site that’s never an issue, but once you start dealing with thousands of pages it becomes a problem.

The solution is to split everything into multiple sitemap files, one for articles, one for products, one for state pages, and then create a single index file that points to all of them: That’s the sitemap_index.xml.

Think of it like a folder. The index file is the folder, and the individual sitemaps are the files inside it.
So if you see sitemap_index.xml, it just means the site is large enough to need more than one sitemap file.

What are the types of sitemaps?

There are two types of sitemaps, and they serve very different audiences:

XML Sitemaps

An XML sitemap is just a text file written in XML format. Inside it, you’ll find a list of URLs from your site, and for each one you can include a few extra details, when the page was last updated, how often it changes, and how important it is relative to other pages.

The main audience for this file is search engines. Google and Bing read it to understand what’s on your site and decide what to crawl. Your regular site visitors will never see it, it’s purely a communication tool between your site and search engines.

Sitemaps HTML

An HTML sitemap is the user-facing version, it’s an actual page on the site, usually linked in the footer, that lists out all the main sections and pages in a readable format.

You can also check the site’s footer, some sites link to their sitemap there. Here’s an example from our own site:

As you can see in the example above, the Sitemap link sits in the footer under the Company column, exactly where users and search engines expect to find it.

Unlike the XML version which is just for search engines, this one is built for real people. If someone lands on your site and can’t find what they’re looking for, a well-structured HTML sitemap gives them a quick overview of everything available.

For an HTML sitemap, you don’t always need to add every single URL from the website. An HTML sitemap should include the important pages users may want to find, such as main service pages, category pages, key blog posts, contact/about pages, and other useful landing pages.

They’re less common than they used to be, but for large sites with complex navigation they’re still worth having. For SEO, the XML sitemap is usually where you include the URLs you want search engines to crawl and index. The HTML sitemap is more for users and site navigation.

Why create an XML sitemap?

So why does a sitemap actually matter? Here’s what it does for your site in practice.

Making sure your important pages actually get indexed

Search engines find pages by following links. If a page has few or no links pointing to it, there’s a good chance it never gets crawled. A sitemap fixes that; you are explicitly telling Google which pages exist and which ones matter, so nothing important gets missed

Help search engines understand the site’s structure and index the content better.

A sitemap doesn’t just list URLs, it also gives search engines a sense of how your content is structured. Which pages are most important, how sections relate to each other, and what gets updated regularly. That context helps Google index your content more accurately and show the right pages for the right searches

It shows your site is well maintained

A sitemap that’s kept up-to-date, with new pages added promptly and removed pages taken out, signals to Google that the site is actively managed. It’s a small thing, but it’s part of the overall picture of a healthy, trustworthy site.

What are the benefits of sitemap.xml for SEO?

Using a sitemap offers several benefits for an SEO strategy. Here are some of the key benefits:

Benefit	Explanation
Efficient indexing	An XML sitemap helps search engines discover and index your website’s main pages, reducing the chance that important content is missed.
Faster discovery of updated content	When you add, update, remove, or redirect pages, your sitemap helps search engines identify those changes more efficiently.
Better understanding of site structure	A sitemap gives search engines a clearer view of how your website is organized and how key pages are connected.
Prioritization of important pages	By including only your most important, indexable URLs, you help search engines focus on the pages that matter most for SEO.
Easier error detection	A sitemap can help identify crawling or indexing issues when analyzed through tools like Google Search Console or SEO auditing platforms.
Integration with SEO tools	Many SEO tools use sitemap data to monitor indexing, crawlability, errors, and overall website performance.

What information is included in a sitemap?

A sitemap.xml file is made up of a few simple elements. Most of them are optional — the only one you actually need is the URL itself. The rest just give search engines extra context about each page.
Here’s what each element does:

<loc>

The element stands for “Location.” It indicates the specific URL of a page on the website. The URL of the page. This is the only required element, every URL in your sitemap needs on.

<lastmod>

The date the page was last updated. Search engines use this to decide whether they need to recrawl a page or if nothing has changed since their last visit

<changefreq>

How often the page is expected to change. You can set this to always, hourly, daily, weekly, monthly, yearly or never. Worth knowing, Google doesn’t follow this value strictly, it uses it as a rough guide at best.

<priority>

How important this page is relative to others on your site, on a scale from 0.0 to 1.0. A value of 1.0 means highest priority. One thing to be clear about, this doesn’t affect your ranking in search results, it just helps Google decide which pages to crawl first when it visits your site.

create-a-sitemap”>How to create a sitemap?

There are a few ways to create a sitemap depending on how your site is built:

Let your CMS do it automatically

Most content management systems can generate a sitemap for you. WordPress, Shopify, Wix and others either include this by default or have plugins that handle it. On WordPress, Yoast SEO and Rank Math are the most common options, install either one and the sitemap gets created and updated automatically every time you add or remove a page. This is the easiest approach for most sites.

Use an online sitemap generator

If your site is small and you don’t use a CMS, an online sitemap generator is the quickest option. You enter your URL, the tool crawls your site and spits out a sitemap file you can upload directly to your root directory. Screaming Frog, Sitebulb and XML-Sitemaps.com all do this well.

Create a sitemap manually

For very small sites with a handful of pages, you can write the XML file yourself in any text editor. It’s straightforward, you just list each URL with the relevant elements. That said, manual sitemaps become a maintenance headache fast. The moment you start adding pages regularly and forgetting to update the file, it starts working against you. For anything beyond a small static site, use one of the other two methods.

For sitemaps with dozens of URLs, generate a sitemap automatically

If your site has too many pages to manage manually, there are tools that will crawl your site and generate the sitemap for you. Here are some of the most commonly used ones:

• Yoast SEO
• Google XML Sitemaps
• Screaming Frog
• Online XML Sitemap Generator
• XML-Sitemaps.com
• Slickplan Sitemap Builder
• Inspyder Sitemap Creator
• Dyno Mapper
• Sitebulb
• JetOctopus
• SE Ranking Sitemap Generator
• Rank Math SEO
• All in One SEO
• A1 Sitemap Generator
• WriteMaps
• Octopus.do
• FlowMapp
• PowerMapper
• VisualSitemaps
• Sitechecker Sitemap Generator

These tools typically crawl your website, identify all pages, and generate a complete XML sitemap containing the necessary elements.

What to include in your sitemap.xml?

As a rule, include any page you want Google to index. That typically means your homepage, product and service pages, category pages, blog posts, guides and contact pages. Policy pages like your privacy policy and terms of service are worth including too.

What to leave out

duplicate pages, paginated pages, pages with noindex tags, login pages and anything with thin or duplicate content. Including low-quality pages in your sitemap wastes crawl budget and can actually work against you.

Where to put it

The sitemap.xml file should sit in the root directory of your site, so example.com/sitemap.xml. That’s where search engines expect to find it.

Once it’s there, add a reference to it in your robots.txt file so search engines can find it immediately:

Sitemap: https://www.example.com/sitemap.xml

Simple line, but worth doing, it removes any guesswork about where your sitemap lives.

Which pages should be included in sitemap.xml?

All important pages on your website should be included in sitemap.xml. See some examples below:

• Main pages: All of your website’s core pages, such as the homepage, product/service pages, category pages, blog pages, etc.;
• Content pages: Include all relevant content pages, such as articles, blog posts, guides, tutorials, and other informational resources;
• Product/Service pages: If your website features products or services, ensure that you include the detail pages for these products or services in the sitemap;
• Contact pages: Contact pages, contact forms, or other important communication pages should be included to facilitate contact with site visitors;
• Policy pages: Important policy pages, such as your privacy policy, terms of service, and return policies, should be present in the sitemap.

Where should the Sitemap be placed?

Your sitemap.xml should live at the root of your domain (example.com/sitemap.xml). That’s where search engines look for it by default.

Then add this line to your robots.txt file:

Sitemap: https://www.example.com/sitemap.xml

That way search engines find it immediately without having to guess.

How to submit a sitemap.xml to Google?

The simplest way is through Google Search Console:

Submit a Sitemap in Search Console

• Access Google Search Console and sign in;
• Select your property
• Click Sitemaps in the left menu
• Enter your sitemap URL and hit Submit

That’s it. Google will start processing it and you’ll be able to see any errors directly in Search Console.

Use the Search Console API

If you’re managing multiple sites or want to automate the process, you can submit sitemaps programmatically via the Search Console API. This is really only relevant if you’re a developer or working at scale, for most sites, doing it manually through the interface is perfectly fine.

Use the Ping Tool

Google has a simple ping URL you can use to notify them when your sitemap is updated. Just visit this URL in your browser, replacing the placeholder with your actual sitemap URL:

https://www.google.com/ping?sitemap=https://www.example.com/sitemap.xml

That’s it. No login needed, no setup, just a quick way to tell Google your sitemap has changed.

How do I update a sitemap.xml file?

If you’re using a CMS with a plugin like Yoast or Rank Math, your sitemap updates automatically whenever you publish or remove a page, you don’t need to do anything.

If you built your sitemap manually, you’ll need to open the file, make the changes, adding new URLs, removing old ones, updating the lastmod dates, save it, and re-upload it to your server.

Once updated, submit it again in Google Search Console or use the ping tool to let Google know something has changed. Then keep an eye on Search Console over the next few days to make sure the new pages are getting picked up and there are no errors.

The main thing to remember is that an outdated sitemap is almost worse than no sitemap, if it’s pointing to pages that no longer exist or missing pages that are important, it’s sending Google in the wrong direction.

What are the sitemap best practices for SEO?

If your site has thousands of pages or very different types of content, split your sitemap into separate files rather than cramming everything into one. For example:

Split your sitemap

If your website has a large number of pages or different content sections, it is recommended to split your sitemap into multiple smaller sitemaps.

For example, you can create separate sitemaps for:

• Blog posts
• Product pages
• Categories
• Images
• Videos
• News articles

It’s easier to manage and easier for Google to process. Then use a sitemap index file to point to all of them from one place.

Create an index

If you have multiple sitemap files, create a sitemap index, a single file that lists all of them. Instead of submitting each sitemap separately in Search Console, you just submit the index file, and Google finds everything from there.

For large sites with different content types, this is the cleanest way to keep things organized.

Respect sitemap size limits

Each XML sitemap should contain no more than 50,000 URLs and should not exceed 50MB uncompressed.

If your website exceeds these limits, you should split your sitemap into multiple files and use a sitemap index. These limits are defined by Google and the official Sitemaps protocol.

Canonical URLs

Only include canonical URLs in your sitemap. A canonical URL is the preferred version of a page that you want search engines to index.

Avoid including duplicate URLs, parameter-based URLs, redirected URLs, or alternate versions of the same page unless they are intentionally meant to be indexed.

Include only indexable pages

Your sitemap should only include pages that you want search engines to crawl and index.

Avoid adding URLs that are:

• Blocked by robots.txt
• Marked with noindex
• Redirected
• Broken or returning 404 errors
• Duplicate or low-value pages
• Internal search result pages
• Login, cart, checkout, or account pages

Keep your sitemap updated

Your sitemap should be updated whenever important pages are added, removed, or changed.

For dynamic websites, blogs, ecommerce stores, and news websites, it is best to generate sitemaps automatically through your CMS, SEO plugin, or sitemap generator.

Add your sitemap to robots.txt

You can help search engines discover your sitemap by adding it to your robots.txt file.

Submit your sitemap to search engines

After creating your sitemap, submit it through tools such as:

• Google Search Console
• Bing Webmaster Tools

This helps search engines discover your sitemap faster and allows you to monitor indexing issues.

Use HTTPS URLs

If your website uses HTTPS, make sure all sitemap URLs also use HTTPS.

Avoid mixing HTTP and HTTPS versions, as this can create duplicate content issues and confuse search engines about the preferred version of your pages.

Avoid Redirects and Errors

Every URL in your sitemap should return a valid 200 OK status code.

Do not include URLs that redirect, return 404 errors, are blocked, or require authentication. A clean sitemap helps search engines trust and process your website structure more effectively.

Keep the Sitemap Clean and Focused

A sitemap is not a place to list every possible URL on your website. It should include your most important, indexable, canonical pages.

What are the main problems related to sitemaps.xml?

Sitemaps.xml files can present several common issues that may affect your website’s indexing and visibility in search engines. Some of the main problems related to sitemaps.xml include:

• Formatting errors: If the XML isn’t structured correctly, with missing tags, typos, or wrong syntax, search engines can’t read the file at all. Always validate your sitemap using Google Search Console or an XML validator before submitting it.
• Outdated sitemap: A sitemap that hasn’t been updated in months can point to pages that no longer exist or miss pages that were recently published. Google will still crawl what’s in the file, so if it’s wrong, you’re wasting crawl budget.
• Incorrect or non-indexable URLs: Including pages you shouldn’t, noindex pages, redirect chains, duplicate pages, and low-quality content have no place in a sitemap. Including them tells Google these pages matter — which is the opposite of what you want.
• Overly large or fragmented sitemap: Sitemap too large Google’s limit is 50,000 URLs or 50MB per file. If you’re hitting that, split into multiple sitemaps and use a sitemap index file to tie them together.
• Lack of alignment with actual site content: Sitemap doesn’t match the actual site If your sitemap lists pages that redirect or return errors, Google loses trust in the file over time. Keep it clean, only include pages that are live, indexable and return a 200 status code.”*

Ready to Improve Your SEO?

A sitemap is one of those things that takes an hour to set up properly and then just runs in the background, but getting it wrong can quietly hold back your indexing for months without you realizing it.

Keep it clean, keep it updated, and make sure it only includes pages you actually want Google to crawl. Do that and it’ll do its job.

If you’d like help with your overall SEO strategy, check out our SEO consulting service, or get in touch and we’ll take a look at what your site needs.