Basic robots.txt Example

A minimal robots.txt example for public websites. It allows crawling and points crawlers to the sitemap.

robots.txt 
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

How to use this example

  1. Replace example.com with the canonical host.
  2. Place the file at /robots.txt.
  3. Check it in Search Console after deployment.

Notes

  • robots.txt is public and not a security feature.
  • Use noindex or authentication for content that should not appear in search.

Related tools and fixes

SEO robots.txt Generator

Generate a clean robots.txt file for crawlers, staging blocks, sitemap discovery, and simple SEO crawl control.

 Sitemap Sitemap Checker

Check pasted XML sitemaps for basic structure, URL counts, loc values, lastmod format and common SEO mistakes.

 HTTP HTTP Headers Checker

Analyze pasted HTTP response headers for cache, redirects, security headers and content-type issues without making server-side requests.

 SEO robots.txt Disallow Mistakes Fix

A small Disallow rule can block valuable pages or rendering assets. Review robots.txt as a public crawl-control file, not a security layer.

Last updated: May 18, 2026