robots.txt and XML Sitemaps for Beginners
What each file does, a safe starter version of both, and the mistakes that accidentally hide sites from Google.
SEO, AEO and GEO consultant · Technical SEO, local SEO and content strategy
Last reviewed November 10, 2026
SEO, AEO & GEO consultant based in Rangpur, Bangladesh. Technical SEO, local search and content strategy. Full bio →
robots.txt tells crawlers which parts of your site they may crawl; an XML sitemap lists the URLs you want discovered. robots.txt does not reliably keep a page out of search results — use `noindex` for that — and a sitemap does not guarantee indexing.
A safe starter robots.txt
For most small sites a minimal file is best: allow everything and point to your sitemap.
`User-agent: *` · `Allow: /` · `Sitemap: https://example.com/sitemap.xml`
What a sitemap should contain
- Only canonical, indexable URLs that return 200.
- Accurate `lastmod` dates — only change them when content really changes.
- No redirected, noindex or error pages.
- Most CMSs and SEO plugins generate one automatically (WordPress setup).
Submit and monitor
- Add the sitemap in Search Console under Sitemaps.
- Also submit it in Bing Webmaster Tools.
- Check the Pages report for excluded URLs.
Mistakes that hide sites
- `Disallow: /` left over from a staging site.
- Blocking CSS or JavaScript Google needs to render pages.
- Blocking a page in robots.txt while adding noindex to it — Google can't see the noindex if it can't crawl.
What about AI crawlers?
Some AI crawlers respect robots.txt rules by user-agent. Decide deliberately; see llms.txt explained for the related file.
Sources and limits
This is a practical guide based on public documentation and day-to-day SEO work. It reports no original dataset, guarantees no ranking, and gives no fixed timeline.
Frequently asked
Google says small, well-linked sites may not need one, but it is easy to add and helps with monitoring.
At the root of the domain, for example example.com/robots.txt.
Use a noindex tag and let Google crawl it, or remove the page and return 404/410.