XML sitemaps and robots.txt: what they do and how to get them right
Two small files decide a surprising amount of how search engines treat your site. One tells crawlers where they may go, the other tells them what is worth looking at. Both are easy to set up and easy to get subtly wrong, and a single careless line can hide an entire site from search for weeks. Here is what each file does and how to check yours.
What each file actually does
Robots.txt is a set of crawling instructions that lives at the root of the domain, for example example.com/robots.txt. It tells crawlers which paths they should not request.
Crucially, it controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it, just without a useful description, because the crawler was never allowed to read it.
To keep a page out of the index you need a noindex meta tag on the page itself, and the page must stay crawlable so the tag can be seen. Blocking it in robots.txt and adding noindex at the same time does not work, because the crawler never reads the tag.
An XML sitemap is the opposite: a list of the pages you do want indexed, with the date each was last changed. It is a hint, not a command, but on larger sites it speeds up discovery considerably.
Neither file improves rankings. They affect discovery and access, which is why their impact is invisible when everything is fine and dramatic when something is wrong.
What belongs in a sitemap
The rule is simple: a sitemap should list the canonical, indexable pages you actually want people to find, and nothing else.
- Include the real content pages: home, services, products, categories, articles, location pages.
- Exclude anything with noindex, anything redirected, anything returning an error and anything that canonicalises to another URL.
- Exclude thin utility pages such as thank you pages, internal search results, login screens, cart and checkout.
- Use the exact final URL, with the right protocol, the right host and consistent trailing slashes.
- Keep lastmod honest. If every URL claims it changed today, the signal becomes worthless.
- Split into multiple sitemaps with an index file once you pass a few thousand URLs, or simply to keep products, articles and static pages separate.
For a multilingual site, either list every language version in one sitemap or use hreflang annotations inside it. The important part is that each language URL appears exactly once, in the form you consider canonical.
Getting robots.txt right
Most sites need a very short robots.txt. Complexity here creates risk without adding value.
A sensible baseline allows all crawlers, blocks the handful of paths that genuinely should not be crawled, and points to the sitemap with a Sitemap line at the end.
Block things like the admin area, internal search result pages, filter and sort parameters that generate endless combinations, and staging paths that live on the same domain.
Do not block CSS, JavaScript or image directories. Search engines render pages and need those files to see the layout the way a visitor does. Blocking them can make a perfectly good page look broken.
Remember that robots.txt is public. Anyone can read it, so it is not a place to hide sensitive directories. If something must stay private, protect it with authentication.
If you need to control AI crawlers separately, you can add rules for specific user agents. Decide deliberately whether you want your content used for training or quoted by AI assistants, because both directions are defensible.
Submitting and monitoring
Add the sitemap address to robots.txt and submit it in Google Search Console and Bing Webmaster Tools. Submission is a one off action, but monitoring is not.
In Search Console, the sitemap report shows how many URLs were discovered and when the file was last read. If the number is far below the real page count, the sitemap is probably incomplete or partly invalid.
The page indexing report is more useful still. It groups URLs by why they are not indexed: blocked by robots.txt, marked noindex, duplicate without a user selected canonical, crawled but not indexed.
Pay attention to the difference between discovered and crawled. Pages that stay discovered for a long time usually mean the site has crawl budget issues, slow responses or too many low value URLs.
Modern platforms generate sitemaps automatically, which is good, but check what they generate. Default configurations often include tag archives, author pages and paginated lists that dilute the file.
Bing also supports IndexNow, which pushes changed URLs immediately instead of waiting for the next crawl. It is cheap to implement and useful for sites that publish frequently.
Mistakes that quietly break things
The most expensive mistake is a development robots.txt going live. A single Disallow line covering the whole site, copied from staging, can remove a site from search within days.
Almost as common is blocking a path in robots.txt while expecting the noindex tag on those pages to work. The two cancel each other out and the URLs linger in the index without descriptions.
Another frequent problem is a sitemap full of redirects and dead URLs after a redesign, usually because the file was generated once and never regenerated.
Watch for mixed URL formats too: some entries with www, some without, some with a trailing slash, some without. Each variation invites duplicate handling.
Check both files after every migration, redesign or platform change, and add them to your launch checklist. It takes two minutes to open example.com/robots.txt and example.com/sitemap.xml in a browser.
If traffic drops suddenly and inexplicably, these two files are the first place to look, before anyone starts rewriting content or blaming an algorithm update.
FAQ
Does every website need a sitemap?+
Not strictly. A small site with clear internal linking, where every page is reachable in a few clicks from the home page, will be discovered perfectly well without one. Sitemaps become genuinely useful when a site is large, when pages are added frequently, when parts of the site are poorly linked internally, or when a new domain has few external links and needs help getting discovered. They also help for content that changes often, because the lastmod date signals when a page is worth recrawling. Since virtually every modern platform generates a sitemap automatically, there is little reason to skip it. The important part is not having a sitemap but having an accurate one, containing only canonical, indexable URLs that return a successful response.
Can robots.txt remove a page from Google?+
No, and this is the most common misunderstanding about the file. Robots.txt stops compliant crawlers from requesting a page, but a URL that is blocked can still appear in search results if other pages link to it, typically shown without a description because the crawler was never allowed to read the content. To remove a page from the index you need a noindex meta tag or HTTP header on the page, and the page must remain crawlable so the instruction can actually be read. If the URL is already blocked, unblock it first, let the crawler see the noindex tag, and only consider blocking it later once it has dropped out. For urgent cases, Search Console has a removal tool that hides a URL temporarily while the permanent fix takes effect.
Should I block AI crawlers in robots.txt?+
It is a business decision rather than an SEO one. Blocking AI training crawlers protects your content from being used to train models, which matters if your text is a core asset such as original research, courses or paid material. The trade off is visibility: assistants that cannot read your site cannot cite it, and a growing share of people now ask AI tools for recommendations instead of browsing search results. Many sites take a middle position, allowing crawlers that provide citations and live answers while blocking pure training crawlers, which requires listing the specific user agents you want to treat differently. Whatever you choose, document the decision, because these rules are easy to forget and tend to be discovered years later by someone wondering why the site is invisible to a popular assistant.